[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:38:43 dist-test-slave-dc95 krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:38:43 dist-test-slave-dc95 krb5kdc[399](info): set up 2 sockets
Sep 25 04:38:43 dist-test-slave-dc95 krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260925 04:38:44.472059 392 mini_kdc.cc:121] Time spent starting KDC: real 0.952s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/krb5kdc/test-user.keytab.
Sep 25 04:38:44 dist-test-slave-dc95 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:38:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:38:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 04:38:44.598887 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43081
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43081
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:38:44.712036 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:38:44.712333 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:38:44.712378 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:38:44.716553 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:38:44.716632 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:38:44.716650 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:38:44.716666 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:38:44.716681 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:38:44.722193 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43081
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43081
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:38:44.723312 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:38:44.724480 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:38:44.734123 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:38:44.734155 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:38:44.734169 412 server_base.cc:1034] running on GCE node
W20260925 04:38:44.734118 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:38:44.735163 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:38:44.736528 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:38:44.737708 412 hybrid_clock.cc:648] HybridClock initialized: now 1790311124737694 us; error 43 us; skew 500 ppm
Sep 25 04:38:44 dist-test-slave-dc95 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311124, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:38:44.740635 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 04:38:44.741642 412 webserver.cc:460] Webserver started at http://127.0.98.62:37845/ using document root <none> and password file <none>
I20260925 04:38:44.742203 412 fs_manager.cc:359] Metadata directory not provided
I20260925 04:38:44.742266 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:38:44.742453 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:38:44.744715 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/minicluster-data/master-0/data/instance:
uuid: "9fa9e01879f94ac8af2253f083ad414e"
format_stamp: "Formatted at 2026-09-25 04:38:44 on dist-test-slave-dc95"
server_key: "cab2f012496ef43ea0964d1d8b42dbe1"
server_key_iv: "56c0f20055cfcb0bff8af888b1989316"
server_key_version: "encryptionkey@0"
I20260925 04:38:44.745312 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/minicluster-data/master-0/wal/instance:
uuid: "9fa9e01879f94ac8af2253f083ad414e"
format_stamp: "Formatted at 2026-09-25 04:38:44 on dist-test-slave-dc95"
server_key: "cab2f012496ef43ea0964d1d8b42dbe1"
server_key_iv: "56c0f20055cfcb0bff8af888b1989316"
server_key_version: "encryptionkey@0"
I20260925 04:38:44.749385 412 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 04:38:44.752216 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:38:44.753682 412 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:38:44.753813 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/minicluster-data/master-0/wal
uuid: "9fa9e01879f94ac8af2253f083ad414e"
format_stamp: "Formatted at 2026-09-25 04:38:44 on dist-test-slave-dc95"
server_key: "cab2f012496ef43ea0964d1d8b42dbe1"
server_key_iv: "56c0f20055cfcb0bff8af888b1989316"
server_key_version: "encryptionkey@0"
I20260925 04:38:44.753929 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:38:44.766319 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:38:44.769536 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:38:44.769712 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:38:44.777118 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43081
I20260925 04:38:44.777169 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43081 every 8 connection(s)
I20260925 04:38:44.778200 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
I20260925 04:38:44.781383 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:38:44.785631 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 412
I20260925 04:38:44.785828 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/minicluster-data/master-0/wal/instance
I20260925 04:38:44.787169 392 external_mini_cluster.cc:1428] Setting key e098da386344de148abc6737a168f1cb
I20260925 04:38:44.787770 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fa9e01879f94ac8af2253f083ad414e: Bootstrap starting.
I20260925 04:38:44.790717 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9fa9e01879f94ac8af2253f083ad414e: Neither blocks nor log segments found. Creating new log.
I20260925 04:38:44.791682 480 log.cc:824] T 00000000000000000000000000000000 P 9fa9e01879f94ac8af2253f083ad414e: Log is configured to *not* fsync() on all Append() calls
I20260925 04:38:44.794317 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fa9e01879f94ac8af2253f083ad414e: No bootstrap required, opened a new log
I20260925 04:38:44.799382 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9fa9e01879f94ac8af2253f083ad414e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa9e01879f94ac8af2253f083ad414e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43081 } }
I20260925 04:38:44.799726 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9fa9e01879f94ac8af2253f083ad414e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:38:44.799788 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9fa9e01879f94ac8af2253f083ad414e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fa9e01879f94ac8af2253f083ad414e, State: Initialized, Role: FOLLOWER
I20260925 04:38:44.800390 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9fa9e01879f94ac8af2253f083ad414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa9e01879f94ac8af2253f083ad414e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43081 } }
I20260925 04:38:44.800566 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9fa9e01879f94ac8af2253f083ad414e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:38:44.800633 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9fa9e01879f94ac8af2253f083ad414e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:38:44.800735 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9fa9e01879f94ac8af2253f083ad414e [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:38:44.801752 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9fa9e01879f94ac8af2253f083ad414e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa9e01879f94ac8af2253f083ad414e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43081 } }
I20260925 04:38:44.802142 480 leader_election.cc:302] T 00000000000000000000000000000000 P 9fa9e01879f94ac8af2253f083ad414e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9fa9e01879f94ac8af2253f083ad414e; no voters:
I20260925 04:38:44.802467 480 leader_election.cc:288] T 00000000000000000000000000000000 P 9fa9e01879f94ac8af2253f083ad414e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:38:44.802587 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9fa9e01879f94ac8af2253f083ad414e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:38:44.802810 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9fa9e01879f94ac8af2253f083ad414e [term 1 LEADER]: Becoming Leader. State: Replica: 9fa9e01879f94ac8af2253f083ad414e, State: Running, Role: LEADER
I20260925 04:38:44.803143 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9fa9e01879f94ac8af2253f083ad414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa9e01879f94ac8af2253f083ad414e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43081 } }
I20260925 04:38:44.803851 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9fa9e01879f94ac8af2253f083ad414e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:38:44.805641 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fa9e01879f94ac8af2253f083ad414e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9fa9e01879f94ac8af2253f083ad414e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa9e01879f94ac8af2253f083ad414e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43081 } } }
I20260925 04:38:44.805778 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fa9e01879f94ac8af2253f083ad414e [sys.catalog]: This master's current role is: LEADER
I20260925 04:38:44.805949 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fa9e01879f94ac8af2253f083ad414e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fa9e01879f94ac8af2253f083ad414e. Latest consensus state: current_term: 1 leader_uuid: "9fa9e01879f94ac8af2253f083ad414e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa9e01879f94ac8af2253f083ad414e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43081 } } }
I20260925 04:38:44.806069 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fa9e01879f94ac8af2253f083ad414e [sys.catalog]: This master's current role is: LEADER
I20260925 04:38:44.806281 500 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 25 04:38:44 dist-test-slave-dc95 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:38:44.809700 500 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:38:44.815876 500 catalog_manager.cc:1347] Generated new cluster ID: c9a74561cd8b40959f5760c3512c849f
I20260925 04:38:44.815948 500 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:38:44.816290 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:44.790597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37666 (local address 127.0.98.62:43081)
0925 04:38:44.791132 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:44.791151 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:44.792420 (+ 1269us) server_negotiation.cc:407] Connection header received
0925 04:38:44.797137 (+ 4717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:44.797158 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:44.797538 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:44.797857 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:44.801093 (+ 3236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:44.802222 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:44.804047 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:44.804781 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:44.804960 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:44.809687 (+ 4727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:38:44.809695 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:38:44.809721 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:38:44.809761 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:38:44.811920 (+ 2159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:44.812625 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:44.812631 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:44.812638 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:44.812807 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:44.813198 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:44.813201 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:44.813203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:44.813745 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:38:44.814458 (+ 713us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:44.814947 (+ 489us) server_negotiation.cc:299] Negotiation successful
0925 04:38:44.815265 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":170,"threads_started":1}
I20260925 04:38:44.825429 500 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:38:44.826412 500 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:38:44.834259 500 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9fa9e01879f94ac8af2253f083ad414e: Generated new TSK 0
I20260925 04:38:44.835088 500 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 04:38:44.871876 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43081
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:38:45.008790 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:38:45.009083 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:38:45.009128 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:38:45.013217 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:38:45.013285 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:38:45.013348 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 04:38:45.018790 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43081
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:38:45.019861 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:38:45.020941 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:38:45.031342 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:38:45.031278 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:38:45.031306 506 server_base.cc:1034] running on GCE node
W20260925 04:38:45.031258 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:38:45.032070 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:38:45.032838 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:38:45.034019 506 hybrid_clock.cc:648] HybridClock initialized: now 1790311125033999 us; error 38 us; skew 500 ppm
Sep 25 04:38:45 dist-test-slave-dc95 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311125, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:38:45.037271 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 04:38:45.038319 506 webserver.cc:460] Webserver started at http://127.0.98.1:43451/ using document root <none> and password file <none>
I20260925 04:38:45.038908 506 fs_manager.cc:359] Metadata directory not provided
I20260925 04:38:45.038982 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:38:45.039193 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:38:45.041392 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/minicluster-data/ts-0/data/instance:
uuid: "e280ec29c59a42f88b2c9183d0d95477"
format_stamp: "Formatted at 2026-09-25 04:38:45 on dist-test-slave-dc95"
server_key: "87935fcaa09d5d51c191ae928c115e3f"
server_key_iv: "c06be038045f8116db02992d07ab249b"
server_key_version: "encryptionkey@0"
I20260925 04:38:45.042001 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e280ec29c59a42f88b2c9183d0d95477"
format_stamp: "Formatted at 2026-09-25 04:38:45 on dist-test-slave-dc95"
server_key: "87935fcaa09d5d51c191ae928c115e3f"
server_key_iv: "c06be038045f8116db02992d07ab249b"
server_key_version: "encryptionkey@0"
I20260925 04:38:45.045945 506 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260925 04:38:45.048636 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:38:45.050172 506 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:38:45.050308 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/minicluster-data/ts-0/wal
uuid: "e280ec29c59a42f88b2c9183d0d95477"
format_stamp: "Formatted at 2026-09-25 04:38:45 on dist-test-slave-dc95"
server_key: "87935fcaa09d5d51c191ae928c115e3f"
server_key_iv: "c06be038045f8116db02992d07ab249b"
server_key_version: "encryptionkey@0"
I20260925 04:38:45.050431 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:38:45.067154 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:38:45.070729 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:38:45.070948 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:38:45.071595 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:38:45.072500 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:38:45.072548 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:38:45.072602 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:38:45.072631 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:38:45.081364 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40413
I20260925 04:38:45.081393 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40413 every 8 connection(s)
I20260925 04:38:45.082270 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
I20260925 04:38:45.089279 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 506
I20260925 04:38:45.089382 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/minicluster-data/ts-0/wal/instance
I20260925 04:38:45.089627 392 external_mini_cluster.cc:1428] Setting key adb975e08ab7777bebbb84b8a63b7415
Sep 25 04:38:45 dist-test-slave-dc95 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311125, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:38:45.096643 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:45.084247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38669 (local address 127.0.98.62:43081)
0925 04:38:45.084437 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:45.084442 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:45.085318 (+ 876us) server_negotiation.cc:407] Connection header received
0925 04:38:45.086687 (+ 1369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:45.086690 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:45.086747 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:45.086833 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:45.088200 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:45.088779 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:45.090061 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:45.090340 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:45.090446 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:45.093143 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:38:45.093146 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:38:45.093148 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:38:45.093174 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:38:45.094549 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:45.095062 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:45.095064 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:45.095066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:45.095148 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:45.095475 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:45.095478 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:45.095480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:45.095719 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:38:45.095807 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:45.096333 (+ 526us) server_negotiation.cc:299] Negotiation successful
0925 04:38:45.096473 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 04:38:45.097514 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:45.084690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43081 (local address 127.0.98.1:38669)
0925 04:38:45.085160 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:45.085204 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 04:38:45.086444 (+ 1240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:38:45.087014 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:38:45.087022 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:38:45.087444 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:38:45.088045 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:45.088062 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:45.088914 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:45.088917 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:45.089950 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:45.089960 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:45.090503 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:45.090508 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:45.090687 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:45.091269 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:38:45.091286 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:38:45.092962 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:38:45.094662 (+ 1700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:45.094668 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:45.094686 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:45.094962 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:45.095230 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:45.095233 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:45.095235 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:45.095379 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:45.095817 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:38:45.095825 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:38:45.096097 (+ 272us) client_negotiation.cc:769] Sending connection context
0925 04:38:45.096276 (+ 179us) client_negotiation.cc:240] Negotiation successful
0925 04:38:45.096639 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":120,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 04:38:45.098933 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43081
I20260925 04:38:45.099152 635 heartbeater.cc:464] Registering TS with master...
I20260925 04:38:45.099720 635 heartbeater.cc:511] Master 127.0.98.62:43081 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260925 04:38:45.101329 444 ts_manager.cc:194] Registered new tserver with Master: e280ec29c59a42f88b2c9183d0d95477 (127.0.98.1:40413)
I20260925 04:38:45.102744 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38669
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 04:38:45.118815 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43081
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:38:45.232110 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:38:45.232390 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:38:45.232450 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:38:45.236531 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:38:45.236603 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:38:45.236685 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 04:38:45.242026 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43081
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:38:45.243137 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:38:45.244257 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:38:45.254288 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:38:45.254395 640 server_base.cc:1034] running on GCE node
W20260925 04:38:45.254237 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:38:45.254236 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:38:45.255031 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:38:45.255860 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:38:45.257061 640 hybrid_clock.cc:648] HybridClock initialized: now 1790311125257038 us; error 42 us; skew 500 ppm
Sep 25 04:38:45 dist-test-slave-dc95 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311125, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:38:45.260089 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 04:38:45.261029 640 webserver.cc:460] Webserver started at http://127.0.98.2:44911/ using document root <none> and password file <none>
I20260925 04:38:45.261595 640 fs_manager.cc:359] Metadata directory not provided
I20260925 04:38:45.261662 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:38:45.261878 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:38:45.264415 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/minicluster-data/ts-1/data/instance:
uuid: "fa5341ae59db4fe28d5ceefb76f1c9fa"
format_stamp: "Formatted at 2026-09-25 04:38:45 on dist-test-slave-dc95"
server_key: "28e42db2dac1776bc5d391559400cdcb"
server_key_iv: "75e068c8dbc448c885cdef99c5d8642b"
server_key_version: "encryptionkey@0"
I20260925 04:38:45.265074 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/minicluster-data/ts-1/wal/instance:
uuid: "fa5341ae59db4fe28d5ceefb76f1c9fa"
format_stamp: "Formatted at 2026-09-25 04:38:45 on dist-test-slave-dc95"
server_key: "28e42db2dac1776bc5d391559400cdcb"
server_key_iv: "75e068c8dbc448c885cdef99c5d8642b"
server_key_version: "encryptionkey@0"
I20260925 04:38:45.269476 640 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:38:45.272670 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:38:45.274151 640 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 04:38:45.274276 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/minicluster-data/ts-1/wal
uuid: "fa5341ae59db4fe28d5ceefb76f1c9fa"
format_stamp: "Formatted at 2026-09-25 04:38:45 on dist-test-slave-dc95"
server_key: "28e42db2dac1776bc5d391559400cdcb"
server_key_iv: "75e068c8dbc448c885cdef99c5d8642b"
server_key_version: "encryptionkey@0"
I20260925 04:38:45.274397 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:38:45.286875 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:38:45.291164 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:38:45.291330 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:38:45.291951 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:38:45.292822 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:38:45.292882 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:38:45.292924 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:38:45.292939 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:38:45.304418 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37651
I20260925 04:38:45.304466 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37651 every 8 connection(s)
I20260925 04:38:45.305474 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
I20260925 04:38:45.305655 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 640
I20260925 04:38:45.305763 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/minicluster-data/ts-1/wal/instance
I20260925 04:38:45.306061 392 external_mini_cluster.cc:1428] Setting key 02ce0798f0eb5d41eff9bb7fbe2ae7e1
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Sep 25 04:38:45 dist-test-slave-dc95 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311125, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:38:45.324525 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:45.307549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55693 (local address 127.0.98.62:43081)
0925 04:38:45.307696 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:45.307701 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:45.309139 (+ 1438us) server_negotiation.cc:407] Connection header received
0925 04:38:45.310888 (+ 1749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:45.310893 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:45.310954 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:45.311042 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:45.312386 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:45.313009 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:45.314719 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:45.314978 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:45.315070 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:45.320310 (+ 5240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:38:45.320315 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:38:45.320319 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:38:45.320354 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:38:45.322018 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:45.322626 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:45.322630 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:45.322632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:45.322732 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:45.323149 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:45.323153 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:45.323155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:45.323406 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:38:45.323515 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:45.324201 (+ 686us) server_negotiation.cc:299] Negotiation successful
0925 04:38:45.324345 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 04:38:45.325730 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:45.308306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43081 (local address 127.0.98.2:55693)
0925 04:38:45.308960 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:45.309016 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 04:38:45.310580 (+ 1564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:38:45.311213 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:38:45.311221 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:38:45.311597 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:38:45.312224 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:45.312241 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:45.313150 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:45.313154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:45.314575 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:45.314586 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:45.315151 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:45.315155 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:45.315941 (+ 786us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:45.316691 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:38:45.316711 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:38:45.319995 (+ 3284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:38:45.322172 (+ 2177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:45.322178 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:45.322195 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:45.322507 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:45.322859 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:45.322862 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:45.322865 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:45.323036 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:45.323534 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:38:45.323541 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:38:45.323932 (+ 391us) client_negotiation.cc:769] Sending connection context
0925 04:38:45.324105 (+ 173us) client_negotiation.cc:240] Negotiation successful
0925 04:38:45.324399 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":152,"threads_started":1}
I20260925 04:38:45.327401 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43081
I20260925 04:38:45.327828 769 heartbeater.cc:464] Registering TS with master...
I20260925 04:38:45.328545 769 heartbeater.cc:511] Master 127.0.98.62:43081 requested a full tablet report, sending...
I20260925 04:38:45.329741 444 ts_manager.cc:194] Registered new tserver with Master: fa5341ae59db4fe28d5ceefb76f1c9fa (127.0.98.2:37651)
I20260925 04:38:45.330420 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55693
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 04:38:45.339112 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43081
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:38:45.452849 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:38:45.453130 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:38:45.453176 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:38:45.457465 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:38:45.457542 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:38:45.457630 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 04:38:45.463114 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43081
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:38:45.464280 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:38:45.465341 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:38:45.475298 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:38:45.475456 774 server_base.cc:1034] running on GCE node
W20260925 04:38:45.475312 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:38:45.475308 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:38:45.476203 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:38:45.476907 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:38:45.478085 774 hybrid_clock.cc:648] HybridClock initialized: now 1790311125478060 us; error 41 us; skew 500 ppm
Sep 25 04:38:45 dist-test-slave-dc95 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311125, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:38:45.481003 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 04:38:45.481995 774 webserver.cc:460] Webserver started at http://127.0.98.3:41847/ using document root <none> and password file <none>
I20260925 04:38:45.482574 774 fs_manager.cc:359] Metadata directory not provided
I20260925 04:38:45.482650 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:38:45.482854 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:38:45.485083 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/minicluster-data/ts-2/data/instance:
uuid: "eb8aafb0e43a4fd3876389ab0ad119c8"
format_stamp: "Formatted at 2026-09-25 04:38:45 on dist-test-slave-dc95"
server_key: "3fa92cd931f55f08ca080597f5c38ac9"
server_key_iv: "057c0f377c02f53ba8aa0ccf8ee383be"
server_key_version: "encryptionkey@0"
I20260925 04:38:45.485658 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/minicluster-data/ts-2/wal/instance:
uuid: "eb8aafb0e43a4fd3876389ab0ad119c8"
format_stamp: "Formatted at 2026-09-25 04:38:45 on dist-test-slave-dc95"
server_key: "3fa92cd931f55f08ca080597f5c38ac9"
server_key_iv: "057c0f377c02f53ba8aa0ccf8ee383be"
server_key_version: "encryptionkey@0"
I20260925 04:38:45.489831 774 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260925 04:38:45.492630 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:38:45.494174 774 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:38:45.494323 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/minicluster-data/ts-2/wal
uuid: "eb8aafb0e43a4fd3876389ab0ad119c8"
format_stamp: "Formatted at 2026-09-25 04:38:45 on dist-test-slave-dc95"
server_key: "3fa92cd931f55f08ca080597f5c38ac9"
server_key_iv: "057c0f377c02f53ba8aa0ccf8ee383be"
server_key_version: "encryptionkey@0"
I20260925 04:38:45.494451 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:38:45.507534 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:38:45.511032 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:38:45.511238 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:38:45.511942 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:38:45.513034 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:38:45.513099 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:38:45.513157 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:38:45.513188 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:38:45.522843 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39417
I20260925 04:38:45.522859 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39417 every 8 connection(s)
I20260925 04:38:45.523785 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
I20260925 04:38:45.525599 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 774
I20260925 04:38:45.525712 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311123478858-392-0/minicluster-data/ts-2/wal/instance
I20260925 04:38:45.525987 392 external_mini_cluster.cc:1428] Setting key 158306f31bdf7522e0222fbddfe9a0e3
Sep 25 04:38:45 dist-test-slave-dc95 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311125, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:38:45.537940 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:45.525624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35901 (local address 127.0.98.62:43081)
0925 04:38:45.525803 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:45.525807 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:45.526715 (+ 908us) server_negotiation.cc:407] Connection header received
0925 04:38:45.528119 (+ 1404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:45.528122 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:45.528176 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:45.528249 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:45.529470 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:45.529995 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:45.531368 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:45.531598 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:45.531731 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:45.534305 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:38:45.534309 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:38:45.534311 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:38:45.534340 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:38:45.535745 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:45.536298 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:45.536302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:45.536304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:45.536396 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:45.536811 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:45.536814 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:45.536816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:45.537015 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:38:45.537101 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:45.537646 (+ 545us) server_negotiation.cc:299] Negotiation successful
0925 04:38:45.537808 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260925 04:38:45.538797 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:45.526071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43081 (local address 127.0.98.3:35901)
0925 04:38:45.526564 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:45.526616 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 04:38:45.527908 (+ 1292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:38:45.528447 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:38:45.528454 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:38:45.528835 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:38:45.529316 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:45.529339 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:45.530113 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:45.530116 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:45.531241 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:45.531252 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:45.531779 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:45.531782 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:45.531952 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:45.532502 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:38:45.532519 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:38:45.534095 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:38:45.535874 (+ 1779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:45.535880 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:45.535894 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:45.536179 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:45.536526 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:45.536529 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:45.536531 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:45.536690 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:45.537127 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:38:45.537133 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:38:45.537400 (+ 267us) client_negotiation.cc:769] Sending connection context
0925 04:38:45.537567 (+ 167us) client_negotiation.cc:240] Negotiation successful
0925 04:38:45.537900 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":115,"threads_started":1}
I20260925 04:38:45.539971 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43081
I20260925 04:38:45.540231 903 heartbeater.cc:464] Registering TS with master...
I20260925 04:38:45.540752 903 heartbeater.cc:511] Master 127.0.98.62:43081 requested a full tablet report, sending...
I20260925 04:38:45.541893 443 ts_manager.cc:194] Registered new tserver with Master: eb8aafb0e43a4fd3876389ab0ad119c8 (127.0.98.3:39417)
I20260925 04:38:45.542515 443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35901
I20260925 04:38:45.552520 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 04:38:45 dist-test-slave-dc95 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311125, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 04:38:45 dist-test-slave-dc95 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311125, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 04:38:45.580684 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:45.569622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46250 (local address 127.0.98.1:40413)
0925 04:38:45.569973 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:45.569986 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:45.570386 (+ 400us) server_negotiation.cc:407] Connection header received
0925 04:38:45.570518 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:45.570523 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:45.570729 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:45.570858 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:45.571497 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:45.572388 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:45.573267 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:45.573697 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:45.573788 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:45.576852 (+ 3064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:38:45.576866 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:38:45.576882 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:38:45.576915 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:38:45.578648 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:45.579202 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:45.579208 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:45.579210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:45.579305 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:45.579625 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:45.579630 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:45.579632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:45.579923 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:38:45.580132 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:45.580340 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 04:38:45.580503 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":131,"threads_started":1}
W20260925 04:38:45.581776 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46250
Sep 25 04:38:45 dist-test-slave-dc95 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 04:38:45 dist-test-slave-dc95 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 04:38:45.608557 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:45.599357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46262 (local address 127.0.98.1:40413)
0925 04:38:45.599514 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:45.599519 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:45.599677 (+ 158us) server_negotiation.cc:407] Connection header received
0925 04:38:45.599831 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:45.599833 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:45.599879 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:45.599952 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:45.600698 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:45.601528 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:45.602399 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:45.602658 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:45.602790 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:45.605562 (+ 2772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:38:45.605565 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:38:45.605567 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:38:45.605600 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:38:45.606889 (+ 1289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:45.607380 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:45.607383 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:45.607385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:45.607490 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:45.607785 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:45.607788 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:45.607789 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:45.607992 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:38:45.608098 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:45.608317 (+ 219us) server_negotiation.cc:299] Negotiation successful
0925 04:38:45.608428 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 04:38:45.609766 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 506
I20260925 04:38:45.614777 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 640
I20260925 04:38:45.621304 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 774
I20260925 04:38:45.626348 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 412
2026-09-25T04:38:45Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2138 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:38:45 dist-test-slave-dc95 krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:38:45 dist-test-slave-dc95 krb5kdc[917](info): set up 2 sockets
Sep 25 04:38:45 dist-test-slave-dc95 krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260925 04:38:46.599176 392 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/krb5kdc/test-user.keytab.
Sep 25 04:38:46 dist-test-slave-dc95 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:38:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:38:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 04:38:46.723052 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46055
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46055
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:38:46.838640 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:38:46.838951 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:38:46.839020 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:38:46.843199 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:38:46.843273 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:38:46.843300 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:38:46.843322 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:38:46.843341 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:38:46.848846 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46055
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:38:46.849969 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:38:46.850996 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:38:46.859882 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:38:46.859927 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:38:46.859965 930 server_base.cc:1034] running on GCE node
W20260925 04:38:46.859900 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:38:46.860699 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:38:46.861873 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:38:46.863083 930 hybrid_clock.cc:648] HybridClock initialized: now 1790311126863030 us; error 29 us; skew 500 ppm
Sep 25 04:38:46 dist-test-slave-dc95 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311126, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:38:46.865921 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 04:38:46.866863 930 webserver.cc:460] Webserver started at http://127.0.98.62:45753/ using document root <none> and password file <none>
I20260925 04:38:46.867472 930 fs_manager.cc:359] Metadata directory not provided
I20260925 04:38:46.867540 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:38:46.867770 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:38:46.869983 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/minicluster-data/master-0/data/instance:
uuid: "44d5fb5ca0ac448b94996ea9ec2b31eb"
format_stamp: "Formatted at 2026-09-25 04:38:46 on dist-test-slave-dc95"
server_key: "940c3b7001481bc95723289996aa2bdc"
server_key_iv: "29feec7e48d3ec56ce39b6c13395c073"
server_key_version: "encryptionkey@0"
I20260925 04:38:46.870563 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/minicluster-data/master-0/wal/instance:
uuid: "44d5fb5ca0ac448b94996ea9ec2b31eb"
format_stamp: "Formatted at 2026-09-25 04:38:46 on dist-test-slave-dc95"
server_key: "940c3b7001481bc95723289996aa2bdc"
server_key_iv: "29feec7e48d3ec56ce39b6c13395c073"
server_key_version: "encryptionkey@0"
I20260925 04:38:46.874708 930 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 04:38:46.877429 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:38:46.878814 930 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:38:46.878947 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/minicluster-data/master-0/wal
uuid: "44d5fb5ca0ac448b94996ea9ec2b31eb"
format_stamp: "Formatted at 2026-09-25 04:38:46 on dist-test-slave-dc95"
server_key: "940c3b7001481bc95723289996aa2bdc"
server_key_iv: "29feec7e48d3ec56ce39b6c13395c073"
server_key_version: "encryptionkey@0"
I20260925 04:38:46.879115 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:38:46.901754 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:38:46.904762 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:38:46.904953 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:38:46.912174 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46055
I20260925 04:38:46.912231 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46055 every 8 connection(s)
I20260925 04:38:46.913175 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
I20260925 04:38:46.916078 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:38:46.919804 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 930
I20260925 04:38:46.919924 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/minicluster-data/master-0/wal/instance
I20260925 04:38:46.920186 392 external_mini_cluster.cc:1428] Setting key be26115a2b6231e37d0902b3bc8001f6
I20260925 04:38:46.922302 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44d5fb5ca0ac448b94996ea9ec2b31eb: Bootstrap starting.
I20260925 04:38:46.924594 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44d5fb5ca0ac448b94996ea9ec2b31eb: Neither blocks nor log segments found. Creating new log.
I20260925 04:38:46.925328 998 log.cc:824] T 00000000000000000000000000000000 P 44d5fb5ca0ac448b94996ea9ec2b31eb: Log is configured to *not* fsync() on all Append() calls
I20260925 04:38:46.927429 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44d5fb5ca0ac448b94996ea9ec2b31eb: No bootstrap required, opened a new log
Sep 25 04:38:46 dist-test-slave-dc95 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:38:46.931826 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44d5fb5ca0ac448b94996ea9ec2b31eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44d5fb5ca0ac448b94996ea9ec2b31eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46055 } }
I20260925 04:38:46.932117 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44d5fb5ca0ac448b94996ea9ec2b31eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:38:46.932215 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44d5fb5ca0ac448b94996ea9ec2b31eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d5fb5ca0ac448b94996ea9ec2b31eb, State: Initialized, Role: FOLLOWER
I20260925 04:38:46.932744 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44d5fb5ca0ac448b94996ea9ec2b31eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44d5fb5ca0ac448b94996ea9ec2b31eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46055 } }
I20260925 04:38:46.932881 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 44d5fb5ca0ac448b94996ea9ec2b31eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:38:46.932945 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44d5fb5ca0ac448b94996ea9ec2b31eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:38:46.933048 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44d5fb5ca0ac448b94996ea9ec2b31eb [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:38:46.934127 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44d5fb5ca0ac448b94996ea9ec2b31eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44d5fb5ca0ac448b94996ea9ec2b31eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46055 } }
I20260925 04:38:46.934491 998 leader_election.cc:302] T 00000000000000000000000000000000 P 44d5fb5ca0ac448b94996ea9ec2b31eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44d5fb5ca0ac448b94996ea9ec2b31eb; no voters:
I20260925 04:38:46.934829 998 leader_election.cc:288] T 00000000000000000000000000000000 P 44d5fb5ca0ac448b94996ea9ec2b31eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:38:46.934939 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44d5fb5ca0ac448b94996ea9ec2b31eb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:38:46.935141 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44d5fb5ca0ac448b94996ea9ec2b31eb [term 1 LEADER]: Becoming Leader. State: Replica: 44d5fb5ca0ac448b94996ea9ec2b31eb, State: Running, Role: LEADER
I20260925 04:38:46.935501 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44d5fb5ca0ac448b94996ea9ec2b31eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44d5fb5ca0ac448b94996ea9ec2b31eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46055 } }
I20260925 04:38:46.936246 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44d5fb5ca0ac448b94996ea9ec2b31eb [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:38:46.936892 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44d5fb5ca0ac448b94996ea9ec2b31eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44d5fb5ca0ac448b94996ea9ec2b31eb. Latest consensus state: current_term: 1 leader_uuid: "44d5fb5ca0ac448b94996ea9ec2b31eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44d5fb5ca0ac448b94996ea9ec2b31eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46055 } } }
I20260925 04:38:46.937109 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44d5fb5ca0ac448b94996ea9ec2b31eb [sys.catalog]: This master's current role is: LEADER
I20260925 04:38:46.937286 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44d5fb5ca0ac448b94996ea9ec2b31eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "44d5fb5ca0ac448b94996ea9ec2b31eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44d5fb5ca0ac448b94996ea9ec2b31eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46055 } } }
I20260925 04:38:46.937394 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44d5fb5ca0ac448b94996ea9ec2b31eb [sys.catalog]: This master's current role is: LEADER
I20260925 04:38:46.937439 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:46.921671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54322 (local address 127.0.98.62:46055)
0925 04:38:46.922129 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:46.922145 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:46.922181 (+ 36us) server_negotiation.cc:407] Connection header received
0925 04:38:46.922859 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:46.922879 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:46.923248 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:46.923565 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:46.924347 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:46.925918 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:46.927079 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:46.927869 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:46.928037 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:46.931101 (+ 3064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:38:46.931106 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:38:46.931124 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:38:46.931158 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:38:46.933041 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:46.933646 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:46.933653 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:46.933660 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:46.933867 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:46.934244 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:46.934247 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:46.934249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:46.934872 (+ 623us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:38:46.935860 (+ 988us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:46.936029 (+ 169us) server_negotiation.cc:299] Negotiation successful
0925 04:38:46.936375 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":136,"threads_started":1}
I20260925 04:38:46.937819 1012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:38:46.941183 1012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:38:46.947264 1012 catalog_manager.cc:1347] Generated new cluster ID: 19f58602e5974719aa33e350ed878b93
I20260925 04:38:46.947352 1012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:38:46.959646 1012 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:38:46.961100 1012 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:38:46.978147 1012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 44d5fb5ca0ac448b94996ea9ec2b31eb: Generated new TSK 0
I20260925 04:38:46.979125 1012 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 04:38:47.017030 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46055
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:38:47.131268 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:38:47.131577 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:38:47.131668 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:38:47.136401 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:38:47.136479 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:38:47.136557 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 04:38:47.142318 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46055
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:38:47.143515 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:38:47.144691 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:38:47.156441 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:38:47.156513 1024 server_base.cc:1034] running on GCE node
W20260925 04:38:47.156682 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:38:47.156451 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:38:47.157337 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:38:47.158056 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:38:47.159273 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790311127159257 us; error 34 us; skew 500 ppm
Sep 25 04:38:47 dist-test-slave-dc95 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:38:47.162745 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 04:38:47.163906 1024 webserver.cc:460] Webserver started at http://127.0.98.1:45203/ using document root <none> and password file <none>
I20260925 04:38:47.164502 1024 fs_manager.cc:359] Metadata directory not provided
I20260925 04:38:47.164570 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:38:47.164768 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:38:47.166996 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/minicluster-data/ts-0/data/instance:
uuid: "31cfd1cdbe754020bca1d1fac8f31b45"
format_stamp: "Formatted at 2026-09-25 04:38:47 on dist-test-slave-dc95"
server_key: "cc562d9f61f80430e3d8495abab5a083"
server_key_iv: "2963b44552ae9d8ab2181a85e49751c2"
server_key_version: "encryptionkey@0"
I20260925 04:38:47.167651 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/minicluster-data/ts-0/wal/instance:
uuid: "31cfd1cdbe754020bca1d1fac8f31b45"
format_stamp: "Formatted at 2026-09-25 04:38:47 on dist-test-slave-dc95"
server_key: "cc562d9f61f80430e3d8495abab5a083"
server_key_iv: "2963b44552ae9d8ab2181a85e49751c2"
server_key_version: "encryptionkey@0"
I20260925 04:38:47.172040 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 04:38:47.175127 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:38:47.176762 1024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:38:47.176898 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/minicluster-data/ts-0/wal
uuid: "31cfd1cdbe754020bca1d1fac8f31b45"
format_stamp: "Formatted at 2026-09-25 04:38:47 on dist-test-slave-dc95"
server_key: "cc562d9f61f80430e3d8495abab5a083"
server_key_iv: "2963b44552ae9d8ab2181a85e49751c2"
server_key_version: "encryptionkey@0"
I20260925 04:38:47.177037 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:38:47.197405 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:38:47.200985 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:38:47.201169 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:38:47.201804 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:38:47.202713 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:38:47.202762 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:38:47.202816 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:38:47.202845 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:38:47.211789 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40829
I20260925 04:38:47.211833 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40829 every 8 connection(s)
I20260925 04:38:47.212788 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
I20260925 04:38:47.214933 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 1024
I20260925 04:38:47.215040 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/minicluster-data/ts-0/wal/instance
I20260925 04:38:47.215305 392 external_mini_cluster.cc:1428] Setting key e67c07b54bd22e1ac9f26370909f8aa9
Sep 25 04:38:47 dist-test-slave-dc95 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 04:38:47.228413 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:47.214901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43495 (local address 127.0.98.62:46055)
0925 04:38:47.215060 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:47.215065 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:47.216217 (+ 1152us) server_negotiation.cc:407] Connection header received
0925 04:38:47.217952 (+ 1735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:47.217955 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:47.218015 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:47.218103 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:47.219478 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:47.220098 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:47.221401 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:47.221635 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:47.221726 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:47.224671 (+ 2945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:38:47.224674 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:38:47.224677 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:38:47.224705 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:38:47.226038 (+ 1333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:47.226644 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:47.226647 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:47.226648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:47.226738 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:47.227145 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:47.227148 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:47.227149 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:47.227355 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:38:47.227441 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:47.228145 (+ 704us) server_negotiation.cc:299] Negotiation successful
0925 04:38:47.228268 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 04:38:47.229285 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:47.215443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46055 (local address 127.0.98.1:43495)
0925 04:38:47.216037 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:47.216090 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 04:38:47.217684 (+ 1594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:38:47.218409 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:38:47.218417 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:38:47.218803 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:38:47.219316 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:47.219332 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:47.220229 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:47.220233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:47.221272 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:47.221284 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:47.221785 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:47.221789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:47.221967 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:47.222620 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:38:47.222638 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:38:47.224433 (+ 1795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:38:47.226172 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:47.226178 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:47.226195 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:47.226493 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:47.226855 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:47.226857 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:47.226859 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:47.227021 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:47.227479 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:38:47.227486 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:38:47.227851 (+ 365us) client_negotiation.cc:769] Sending connection context
0925 04:38:47.228041 (+ 190us) client_negotiation.cc:240] Negotiation successful
0925 04:38:47.228341 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":192,"threads_started":1}
I20260925 04:38:47.230559 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46055
I20260925 04:38:47.230806 1153 heartbeater.cc:464] Registering TS with master...
I20260925 04:38:47.231372 1153 heartbeater.cc:511] Master 127.0.98.62:46055 requested a full tablet report, sending...
I20260925 04:38:47.233258 962 ts_manager.cc:194] Registered new tserver with Master: 31cfd1cdbe754020bca1d1fac8f31b45 (127.0.98.1:40829)
I20260925 04:38:47.234509 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43495
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260925 04:38:48.238252 1153 heartbeater.cc:503] Master 127.0.98.62:46055 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab.
W20260925 04:38:48.589709 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.2: real 1.373s user 0.000s sys 0.003s
I20260925 04:38:48.590323 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46055
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:38:48.716840 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:38:48.717144 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:38:48.717188 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:38:48.721498 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:38:48.722275 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:38:48.722404 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 04:38:48.728109 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46055
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:38:48.729306 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:38:48.730418 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:38:48.741025 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:38:48.741253 1158 server_base.cc:1034] running on GCE node
W20260925 04:38:48.741078 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:38:48.741096 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:38:48.741982 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:38:48.742774 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:38:48.743983 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790311128743965 us; error 32 us; skew 500 ppm
Sep 25 04:38:48 dist-test-slave-dc95 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311128, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:38:48.747254 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 04:38:48.748405 1158 webserver.cc:460] Webserver started at http://127.0.98.2:44939/ using document root <none> and password file <none>
I20260925 04:38:48.749018 1158 fs_manager.cc:359] Metadata directory not provided
I20260925 04:38:48.749092 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:38:48.749317 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:38:48.751751 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/minicluster-data/ts-1/data/instance:
uuid: "ce1e7b84cabb446e9cc28003226b6d7e"
format_stamp: "Formatted at 2026-09-25 04:38:48 on dist-test-slave-dc95"
server_key: "0297d79af86a18b9b35699b2ce59cada"
server_key_iv: "989352f774400b15f2e5ac2ec4268764"
server_key_version: "encryptionkey@0"
I20260925 04:38:48.752480 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ce1e7b84cabb446e9cc28003226b6d7e"
format_stamp: "Formatted at 2026-09-25 04:38:48 on dist-test-slave-dc95"
server_key: "0297d79af86a18b9b35699b2ce59cada"
server_key_iv: "989352f774400b15f2e5ac2ec4268764"
server_key_version: "encryptionkey@0"
I20260925 04:38:48.757568 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 04:38:48.761914 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:38:48.763738 1158 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 04:38:48.763906 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/minicluster-data/ts-1/wal
uuid: "ce1e7b84cabb446e9cc28003226b6d7e"
format_stamp: "Formatted at 2026-09-25 04:38:48 on dist-test-slave-dc95"
server_key: "0297d79af86a18b9b35699b2ce59cada"
server_key_iv: "989352f774400b15f2e5ac2ec4268764"
server_key_version: "encryptionkey@0"
I20260925 04:38:48.764040 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:38:48.785637 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:38:48.791824 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:38:48.792029 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:38:48.792698 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:38:48.793745 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:38:48.793805 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:38:48.793872 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:38:48.793902 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:38:48.803064 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37939
I20260925 04:38:48.803095 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37939 every 8 connection(s)
I20260925 04:38:48.804139 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
I20260925 04:38:48.808928 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 1158
I20260925 04:38:48.809068 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/minicluster-data/ts-1/wal/instance
I20260925 04:38:48.809355 392 external_mini_cluster.cc:1428] Setting key 28bdfdb0d2403293997cb398e473e0f0
Sep 25 04:38:48 dist-test-slave-dc95 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311128, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260925 04:38:48.823750 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:48.806487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35975 (local address 127.0.98.62:46055)
0925 04:38:48.806774 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:48.806778 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:48.807893 (+ 1115us) server_negotiation.cc:407] Connection header received
0925 04:38:48.809640 (+ 1747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:48.809644 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:48.809706 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:48.809830 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:48.811668 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:48.812253 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:48.814140 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:48.814403 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:48.814505 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:48.818284 (+ 3779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:38:48.818288 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:38:48.818290 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:38:48.818429 (+ 139us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:38:48.820268 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:48.821022 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:48.821026 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:48.821027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:48.821131 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:48.821609 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:48.821613 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:48.821614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:48.821846 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:38:48.821949 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:48.823195 (+ 1246us) server_negotiation.cc:299] Negotiation successful
0925 04:38:48.823401 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":97,"threads_started":1}
I20260925 04:38:48.824331 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:48.806991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46055 (local address 127.0.98.2:35975)
0925 04:38:48.807700 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:48.807753 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 04:38:48.809341 (+ 1588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:38:48.810087 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:38:48.810097 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:38:48.810702 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:38:48.811443 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:48.811469 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:48.812417 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:48.812420 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:48.813958 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:48.813973 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:48.814609 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:48.814613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:48.814892 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:48.815678 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:38:48.815704 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:38:48.817985 (+ 2281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:38:48.820506 (+ 2521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:48.820514 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:48.820532 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:48.820860 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:48.821249 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:48.821270 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:48.821272 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:48.821481 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:48.822086 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:38:48.822096 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:38:48.822539 (+ 443us) client_negotiation.cc:769] Sending connection context
0925 04:38:48.822923 (+ 384us) client_negotiation.cc:240] Negotiation successful
0925 04:38:48.823292 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":132,"threads_started":1}
I20260925 04:38:48.825795 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46055
I20260925 04:38:48.826107 1287 heartbeater.cc:464] Registering TS with master...
I20260925 04:38:48.826773 1287 heartbeater.cc:511] Master 127.0.98.62:46055 requested a full tablet report, sending...
I20260925 04:38:48.828140 962 ts_manager.cc:194] Registered new tserver with Master: ce1e7b84cabb446e9cc28003226b6d7e (127.0.98.2:37939)
I20260925 04:38:48.828796 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35975
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab.
W20260925 04:38:49.591430 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.3: real 0.780s user 0.000s sys 0.004s
I20260925 04:38:49.592031 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46055
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:38:49.709669 1293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:38:49.710001 1293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:38:49.710048 1293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:38:49.714326 1293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:38:49.714500 1293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:38:49.714605 1293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 04:38:49.720446 1293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46055
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:38:49.721598 1293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:38:49.722625 1293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 04:38:49.732555 1293 server_base.cc:1034] running on GCE node
W20260925 04:38:49.732542 1298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:38:49.732718 1299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:38:49.732547 1301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:38:49.733114 1293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:38:49.733758 1293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:38:49.735005 1293 hybrid_clock.cc:648] HybridClock initialized: now 1790311129734973 us; error 45 us; skew 500 ppm
Sep 25 04:38:49 dist-test-slave-dc95 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:38:49.738262 1293 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 04:38:49.739283 1293 webserver.cc:460] Webserver started at http://127.0.98.3:45889/ using document root <none> and password file <none>
I20260925 04:38:49.739926 1293 fs_manager.cc:359] Metadata directory not provided
I20260925 04:38:49.740001 1293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:38:49.740216 1293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:38:49.742420 1293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/minicluster-data/ts-2/data/instance:
uuid: "c0b1259300b34b318d0094028ee55384"
format_stamp: "Formatted at 2026-09-25 04:38:49 on dist-test-slave-dc95"
server_key: "98fa17468ecd5d89ff0b7c7b934d1235"
server_key_iv: "96f7b3f9768455c5889b6eac7fc749d3"
server_key_version: "encryptionkey@0"
I20260925 04:38:49.743006 1293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c0b1259300b34b318d0094028ee55384"
format_stamp: "Formatted at 2026-09-25 04:38:49 on dist-test-slave-dc95"
server_key: "98fa17468ecd5d89ff0b7c7b934d1235"
server_key_iv: "96f7b3f9768455c5889b6eac7fc749d3"
server_key_version: "encryptionkey@0"
I20260925 04:38:49.747102 1293 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 04:38:49.750138 1308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:38:49.751863 1293 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 04:38:49.752017 1293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/minicluster-data/ts-2/wal
uuid: "c0b1259300b34b318d0094028ee55384"
format_stamp: "Formatted at 2026-09-25 04:38:49 on dist-test-slave-dc95"
server_key: "98fa17468ecd5d89ff0b7c7b934d1235"
server_key_iv: "96f7b3f9768455c5889b6eac7fc749d3"
server_key_version: "encryptionkey@0"
I20260925 04:38:49.752154 1293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:38:49.766198 1293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:38:49.769835 1293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:38:49.770030 1293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:38:49.770641 1293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:38:49.771533 1293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:38:49.771584 1293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:38:49.771679 1293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:38:49.771705 1293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:38:49.780980 1293 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42947
I20260925 04:38:49.780992 1421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42947 every 8 connection(s)
I20260925 04:38:49.782214 1293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
I20260925 04:38:49.789057 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 1293
I20260925 04:38:49.789163 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311123478858-392-0/minicluster-data/ts-2/wal/instance
I20260925 04:38:49.789427 392 external_mini_cluster.cc:1428] Setting key b2d03d6ca4e777a3d5215651b967381f
Sep 25 04:38:49 dist-test-slave-dc95 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:38:49.799243 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:49.784907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37803 (local address 127.0.98.62:46055)
0925 04:38:49.785258 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:49.785263 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:49.786713 (+ 1450us) server_negotiation.cc:407] Connection header received
0925 04:38:49.788212 (+ 1499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:49.788216 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:49.788274 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:49.788385 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:49.789814 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:49.790530 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:49.792374 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:49.792649 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:49.792741 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:49.795493 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:38:49.795497 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:38:49.795499 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:38:49.795528 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:38:49.797166 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:49.797765 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:49.797769 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:49.797771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:49.797858 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:49.798173 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:49.798175 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:49.798176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:49.798392 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:38:49.798475 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:49.798973 (+ 498us) server_negotiation.cc:299] Negotiation successful
0925 04:38:49.799094 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":102,"threads_started":1}
I20260925 04:38:49.800161 1425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:49.785439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46055 (local address 127.0.98.3:37803)
0925 04:38:49.786509 (+ 1070us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:49.786570 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 04:38:49.787946 (+ 1376us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:38:49.788576 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:38:49.788584 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:38:49.789016 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:38:49.789646 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:49.789663 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:49.790711 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:49.790714 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:49.792221 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:49.792233 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:49.792833 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:49.792836 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:49.793057 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:49.793724 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:38:49.793743 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:38:49.795270 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:38:49.797324 (+ 2054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:49.797330 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:49.797346 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:49.797667 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:49.797933 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:49.797935 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:49.797937 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:49.798096 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:49.798474 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:38:49.798479 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:38:49.798756 (+ 277us) client_negotiation.cc:769] Sending connection context
0925 04:38:49.798921 (+ 165us) client_negotiation.cc:240] Negotiation successful
0925 04:38:49.799235 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":850,"thread_start_us":174,"threads_started":1}
I20260925 04:38:49.801329 1422 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46055
I20260925 04:38:49.801563 1422 heartbeater.cc:464] Registering TS with master...
I20260925 04:38:49.802062 1422 heartbeater.cc:511] Master 127.0.98.62:46055 requested a full tablet report, sending...
I20260925 04:38:49.803143 961 ts_manager.cc:194] Registered new tserver with Master: c0b1259300b34b318d0094028ee55384 (127.0.98.3:42947)
I20260925 04:38:49.804005 961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37803
I20260925 04:38:49.804216 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:38:49.816002 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:49.808004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45814 (local address 127.0.98.62:46055)
0925 04:38:49.808164 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:49.808169 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:49.808272 (+ 103us) server_negotiation.cc:407] Connection header received
0925 04:38:49.808373 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:49.808375 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:49.808414 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:49.808483 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:49.809254 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:49.809816 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:49.810669 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:49.810903 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:49.810989 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:49.812482 (+ 1493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:38:49.812486 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:38:49.812488 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:38:49.812517 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:38:49.814000 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:49.814526 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:49.814530 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:49.814531 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:49.814639 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:49.814930 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:49.814933 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:49.814935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:49.815175 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:38:49.815316 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:49.815557 (+ 241us) server_negotiation.cc:299] Negotiation successful
0925 04:38:49.815712 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 04:38:49.821723 961 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45814:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:38:49.824905 961 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:38:49.831177 1287 heartbeater.cc:503] Master 127.0.98.62:46055 was elected leader, sending a full tablet report...
I20260925 04:38:49.842540 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:49.836477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48256 (local address 127.0.98.1:40829)
0925 04:38:49.837257 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:49.837264 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:49.837334 (+ 70us) server_negotiation.cc:407] Connection header received
0925 04:38:49.837425 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:49.837429 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:49.837605 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:49.837772 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:38:49.838446 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:49.839538 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:49.841230 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:49.841825 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:49.841931 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:49.842116 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:49.842280 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 04:38:49.842382 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":689,"thread_start_us":105,"threads_started":1}
I20260925 04:38:49.842531 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:49.836320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40829 (local address 127.0.0.1:48256)
0925 04:38:49.837257 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:49.837268 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 04:38:49.837340 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:38:49.838210 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:38:49.838213 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:38:49.838232 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:38:49.838320 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:49.838325 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:49.839750 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:49.839754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:49.841098 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:49.841107 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:49.841959 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:49.841963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:49.842083 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:49.842129 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 04:38:49.842237 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 04:38:49.842377 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":868,"thread_start_us":86,"threads_started":1}
I20260925 04:38:49.842862 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:49.835369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42947 (local address 127.0.0.1:43046)
0925 04:38:49.836437 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:49.836495 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 04:38:49.836647 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:38:49.838532 (+ 1885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:38:49.838535 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:38:49.838553 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:38:49.838659 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:49.838665 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:49.840494 (+ 1829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:49.840497 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:49.841365 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:49.841374 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:49.842475 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:49.842479 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:49.842560 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:49.842583 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 04:38:49.842625 (+ 42us) client_negotiation.cc:240] Negotiation successful
0925 04:38:49.842713 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":870,"mutex_wait_us":58,"thread_start_us":203,"threads_started":1}
I20260925 04:38:49.843838 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:49.835890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43046 (local address 127.0.98.3:42947)
0925 04:38:49.837983 (+ 2093us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:49.837997 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:49.838018 (+ 21us) server_negotiation.cc:407] Connection header received
0925 04:38:49.838090 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:49.838095 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:49.838364 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:49.838808 (+ 444us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:38:49.839073 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:49.840369 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:49.841489 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:49.842335 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:49.843051 (+ 716us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:49.843205 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:49.843256 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 04:38:49.843677 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1104,"thread_start_us":132,"threads_started":1}
I20260925 04:38:49.844304 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:49.835505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37939 (local address 127.0.0.1:47044)
0925 04:38:49.836961 (+ 1456us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:49.836973 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:38:49.837090 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:38:49.837772 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:38:49.837779 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:38:49.837802 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:38:49.837956 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:49.837963 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:49.840485 (+ 2522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:49.840489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:49.842475 (+ 1986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:49.842484 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:49.843911 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:49.843915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:49.844006 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:49.844027 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 04:38:49.844103 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 04:38:49.844175 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1396,"mutex_wait_us":796,"thread_start_us":128,"threads_started":1}
I20260925 04:38:49.844394 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:49.835721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47044 (local address 127.0.98.2:37939)
0925 04:38:49.836893 (+ 1172us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:49.836900 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:49.837085 (+ 185us) server_negotiation.cc:407] Connection header received
0925 04:38:49.837358 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:49.837364 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:49.837605 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:49.837763 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:38:49.838086 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:49.840343 (+ 2257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:49.842602 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:49.843575 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:49.843700 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:49.843834 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:49.844154 (+ 320us) server_negotiation.cc:299] Negotiation successful
0925 04:38:49.844260 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1063,"thread_start_us":77,"threads_started":1}
I20260925 04:38:49.846328 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 60702bbb6cdd400eac661ebf90aea69a (DEFAULT_TABLE table=test-table [id=c5ed9805b7424c98aa0a0a4417ce5ab6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:38:49.846789 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 60702bbb6cdd400eac661ebf90aea69a (DEFAULT_TABLE table=test-table [id=c5ed9805b7424c98aa0a0a4417ce5ab6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:38:49.846856 1356 tablet_service.cc:1467] Processing CreateTablet for tablet 60702bbb6cdd400eac661ebf90aea69a (DEFAULT_TABLE table=test-table [id=c5ed9805b7424c98aa0a0a4417ce5ab6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:38:49.847563 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60702bbb6cdd400eac661ebf90aea69a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:38:49.847821 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60702bbb6cdd400eac661ebf90aea69a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:38:49.847975 1356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60702bbb6cdd400eac661ebf90aea69a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:38:49.854492 1441 tablet_bootstrap.cc:492] T 60702bbb6cdd400eac661ebf90aea69a P ce1e7b84cabb446e9cc28003226b6d7e: Bootstrap starting.
I20260925 04:38:49.856117 1442 tablet_bootstrap.cc:492] T 60702bbb6cdd400eac661ebf90aea69a P 31cfd1cdbe754020bca1d1fac8f31b45: Bootstrap starting.
I20260925 04:38:49.856221 1443 tablet_bootstrap.cc:492] T 60702bbb6cdd400eac661ebf90aea69a P c0b1259300b34b318d0094028ee55384: Bootstrap starting.
I20260925 04:38:49.858881 1442 tablet_bootstrap.cc:654] T 60702bbb6cdd400eac661ebf90aea69a P 31cfd1cdbe754020bca1d1fac8f31b45: Neither blocks nor log segments found. Creating new log.
I20260925 04:38:49.858881 1443 tablet_bootstrap.cc:654] T 60702bbb6cdd400eac661ebf90aea69a P c0b1259300b34b318d0094028ee55384: Neither blocks nor log segments found. Creating new log.
I20260925 04:38:49.857777 1441 tablet_bootstrap.cc:654] T 60702bbb6cdd400eac661ebf90aea69a P ce1e7b84cabb446e9cc28003226b6d7e: Neither blocks nor log segments found. Creating new log.
I20260925 04:38:49.859891 1442 log.cc:824] T 60702bbb6cdd400eac661ebf90aea69a P 31cfd1cdbe754020bca1d1fac8f31b45: Log is configured to *not* fsync() on all Append() calls
I20260925 04:38:49.859930 1441 log.cc:824] T 60702bbb6cdd400eac661ebf90aea69a P ce1e7b84cabb446e9cc28003226b6d7e: Log is configured to *not* fsync() on all Append() calls
I20260925 04:38:49.860140 1443 log.cc:824] T 60702bbb6cdd400eac661ebf90aea69a P c0b1259300b34b318d0094028ee55384: Log is configured to *not* fsync() on all Append() calls
I20260925 04:38:49.862409 1442 tablet_bootstrap.cc:492] T 60702bbb6cdd400eac661ebf90aea69a P 31cfd1cdbe754020bca1d1fac8f31b45: No bootstrap required, opened a new log
I20260925 04:38:49.862603 1443 tablet_bootstrap.cc:492] T 60702bbb6cdd400eac661ebf90aea69a P c0b1259300b34b318d0094028ee55384: No bootstrap required, opened a new log
I20260925 04:38:49.862672 1442 ts_tablet_manager.cc:1397] T 60702bbb6cdd400eac661ebf90aea69a P 31cfd1cdbe754020bca1d1fac8f31b45: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260925 04:38:49.862834 1443 ts_tablet_manager.cc:1397] T 60702bbb6cdd400eac661ebf90aea69a P c0b1259300b34b318d0094028ee55384: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260925 04:38:49.864323 1441 tablet_bootstrap.cc:492] T 60702bbb6cdd400eac661ebf90aea69a P ce1e7b84cabb446e9cc28003226b6d7e: No bootstrap required, opened a new log
I20260925 04:38:49.864650 1441 ts_tablet_manager.cc:1397] T 60702bbb6cdd400eac661ebf90aea69a P ce1e7b84cabb446e9cc28003226b6d7e: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260925 04:38:49.866971 1443 raft_consensus.cc:348] T 60702bbb6cdd400eac661ebf90aea69a P c0b1259300b34b318d0094028ee55384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0b1259300b34b318d0094028ee55384" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42947 } } peers { permanent_uuid: "ce1e7b84cabb446e9cc28003226b6d7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37939 } } peers { permanent_uuid: "31cfd1cdbe754020bca1d1fac8f31b45" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40829 } }
I20260925 04:38:49.867409 1442 raft_consensus.cc:348] T 60702bbb6cdd400eac661ebf90aea69a P 31cfd1cdbe754020bca1d1fac8f31b45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0b1259300b34b318d0094028ee55384" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42947 } } peers { permanent_uuid: "ce1e7b84cabb446e9cc28003226b6d7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37939 } } peers { permanent_uuid: "31cfd1cdbe754020bca1d1fac8f31b45" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40829 } }
I20260925 04:38:49.867383 1443 raft_consensus.cc:374] T 60702bbb6cdd400eac661ebf90aea69a P c0b1259300b34b318d0094028ee55384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:38:49.867722 1443 raft_consensus.cc:729] T 60702bbb6cdd400eac661ebf90aea69a P c0b1259300b34b318d0094028ee55384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0b1259300b34b318d0094028ee55384, State: Initialized, Role: FOLLOWER
I20260925 04:38:49.867761 1442 raft_consensus.cc:374] T 60702bbb6cdd400eac661ebf90aea69a P 31cfd1cdbe754020bca1d1fac8f31b45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:38:49.867851 1442 raft_consensus.cc:729] T 60702bbb6cdd400eac661ebf90aea69a P 31cfd1cdbe754020bca1d1fac8f31b45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31cfd1cdbe754020bca1d1fac8f31b45, State: Initialized, Role: FOLLOWER
I20260925 04:38:49.868377 1443 consensus_queue.cc:260] T 60702bbb6cdd400eac661ebf90aea69a P c0b1259300b34b318d0094028ee55384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0b1259300b34b318d0094028ee55384" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42947 } } peers { permanent_uuid: "ce1e7b84cabb446e9cc28003226b6d7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37939 } } peers { permanent_uuid: "31cfd1cdbe754020bca1d1fac8f31b45" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40829 } }
I20260925 04:38:49.868436 1442 consensus_queue.cc:260] T 60702bbb6cdd400eac661ebf90aea69a P 31cfd1cdbe754020bca1d1fac8f31b45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0b1259300b34b318d0094028ee55384" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42947 } } peers { permanent_uuid: "ce1e7b84cabb446e9cc28003226b6d7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37939 } } peers { permanent_uuid: "31cfd1cdbe754020bca1d1fac8f31b45" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40829 } }
I20260925 04:38:49.869626 1442 ts_tablet_manager.cc:1428] T 60702bbb6cdd400eac661ebf90aea69a P 31cfd1cdbe754020bca1d1fac8f31b45: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260925 04:38:49.870005 1443 ts_tablet_manager.cc:1428] T 60702bbb6cdd400eac661ebf90aea69a P c0b1259300b34b318d0094028ee55384: Time spent starting tablet: real 0.007s user 0.005s sys 0.002s
I20260925 04:38:49.870440 1422 heartbeater.cc:503] Master 127.0.98.62:46055 was elected leader, sending a full tablet report...
I20260925 04:38:49.873404 1441 raft_consensus.cc:348] T 60702bbb6cdd400eac661ebf90aea69a P ce1e7b84cabb446e9cc28003226b6d7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0b1259300b34b318d0094028ee55384" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42947 } } peers { permanent_uuid: "ce1e7b84cabb446e9cc28003226b6d7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37939 } } peers { permanent_uuid: "31cfd1cdbe754020bca1d1fac8f31b45" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40829 } }
I20260925 04:38:49.873798 1441 raft_consensus.cc:374] T 60702bbb6cdd400eac661ebf90aea69a P ce1e7b84cabb446e9cc28003226b6d7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:38:49.873867 1441 raft_consensus.cc:729] T 60702bbb6cdd400eac661ebf90aea69a P ce1e7b84cabb446e9cc28003226b6d7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1e7b84cabb446e9cc28003226b6d7e, State: Initialized, Role: FOLLOWER
I20260925 04:38:49.874401 1441 consensus_queue.cc:260] T 60702bbb6cdd400eac661ebf90aea69a P ce1e7b84cabb446e9cc28003226b6d7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0b1259300b34b318d0094028ee55384" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42947 } } peers { permanent_uuid: "ce1e7b84cabb446e9cc28003226b6d7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37939 } } peers { permanent_uuid: "31cfd1cdbe754020bca1d1fac8f31b45" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40829 } }
I20260925 04:38:49.876654 1441 ts_tablet_manager.cc:1428] T 60702bbb6cdd400eac661ebf90aea69a P ce1e7b84cabb446e9cc28003226b6d7e: Time spent starting tablet: real 0.012s user 0.005s sys 0.001s
W20260925 04:38:49.966105 1154 tablet.cc:2369] T 60702bbb6cdd400eac661ebf90aea69a P 31cfd1cdbe754020bca1d1fac8f31b45: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 04:38:50.033825 1423 tablet.cc:2369] T 60702bbb6cdd400eac661ebf90aea69a P c0b1259300b34b318d0094028ee55384: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 04:38:50.056037 1288 tablet.cc:2369] T 60702bbb6cdd400eac661ebf90aea69a P ce1e7b84cabb446e9cc28003226b6d7e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:38:50.121384 1449 raft_consensus.cc:482] T 60702bbb6cdd400eac661ebf90aea69a P ce1e7b84cabb446e9cc28003226b6d7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:38:50.121601 1449 raft_consensus.cc:504] T 60702bbb6cdd400eac661ebf90aea69a P ce1e7b84cabb446e9cc28003226b6d7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0b1259300b34b318d0094028ee55384" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42947 } } peers { permanent_uuid: "ce1e7b84cabb446e9cc28003226b6d7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37939 } } peers { permanent_uuid: "31cfd1cdbe754020bca1d1fac8f31b45" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40829 } }
I20260925 04:38:50.122859 1449 leader_election.cc:288] T 60702bbb6cdd400eac661ebf90aea69a P ce1e7b84cabb446e9cc28003226b6d7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0b1259300b34b318d0094028ee55384 (127.0.98.3:42947), 31cfd1cdbe754020bca1d1fac8f31b45 (127.0.98.1:40829)
I20260925 04:38:50.127460 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:50.123308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36181 (local address 127.0.98.3:42947)
0925 04:38:50.123506 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:50.123510 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:50.124194 (+ 684us) server_negotiation.cc:407] Connection header received
0925 04:38:50.124320 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:50.124323 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:50.124373 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:50.124429 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:38:50.124744 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:50.125280 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:50.126454 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:50.126870 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:50.126952 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:50.127028 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:50.127253 (+ 225us) server_negotiation.cc:299] Negotiation successful
0925 04:38:50.127330 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 04:38:50.127477 1450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:50.123183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40829 (local address 127.0.98.2:51881)
0925 04:38:50.124113 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:50.124127 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:38:50.124261 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:38:50.124510 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:38:50.124513 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:38:50.124554 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:38:50.124656 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:50.124661 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:50.125393 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:50.125396 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:50.126317 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:50.126327 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:50.127059 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:50.127063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:50.127171 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:50.127192 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 04:38:50.127266 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 04:38:50.127331 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":850,"mutex_wait_us":59,"spinlock_wait_cycles":3584,"thread_start_us":104,"threads_started":1}
I20260925 04:38:50.127501 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:50.123302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51881 (local address 127.0.98.1:40829)
0925 04:38:50.123548 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:50.123554 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:50.124252 (+ 698us) server_negotiation.cc:407] Connection header received
0925 04:38:50.124360 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:50.124363 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:50.124413 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:50.124487 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:38:50.124767 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:50.125280 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:50.126515 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:50.126931 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:50.127051 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:50.127115 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:50.127296 (+ 181us) server_negotiation.cc:299] Negotiation successful
0925 04:38:50.127390 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154}
I20260925 04:38:50.127686 1451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:50.123183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42947 (local address 127.0.98.2:36181)
0925 04:38:50.124079 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:50.124093 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:38:50.124212 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:38:50.124461 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:38:50.124464 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:38:50.124480 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:38:50.124632 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:50.124637 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:50.125392 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:50.125396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:50.126320 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:50.126330 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:50.126992 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:50.126996 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:50.127092 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:50.127116 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 04:38:50.127186 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 04:38:50.127255 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":831,"mutex_wait_us":93,"spinlock_wait_cycles":24448,"thread_start_us":137,"threads_started":1}
I20260925 04:38:50.128186 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60702bbb6cdd400eac661ebf90aea69a" candidate_uuid: "ce1e7b84cabb446e9cc28003226b6d7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31cfd1cdbe754020bca1d1fac8f31b45" is_pre_election: true
I20260925 04:38:50.128508 1107 raft_consensus.cc:2454] T 60702bbb6cdd400eac661ebf90aea69a P 31cfd1cdbe754020bca1d1fac8f31b45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce1e7b84cabb446e9cc28003226b6d7e in term 0.
I20260925 04:38:50.128799 1376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60702bbb6cdd400eac661ebf90aea69a" candidate_uuid: "ce1e7b84cabb446e9cc28003226b6d7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0b1259300b34b318d0094028ee55384" is_pre_election: true
I20260925 04:38:50.129125 1376 raft_consensus.cc:2454] T 60702bbb6cdd400eac661ebf90aea69a P c0b1259300b34b318d0094028ee55384 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce1e7b84cabb446e9cc28003226b6d7e in term 0.
I20260925 04:38:50.129125 1176 leader_election.cc:302] T 60702bbb6cdd400eac661ebf90aea69a P ce1e7b84cabb446e9cc28003226b6d7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31cfd1cdbe754020bca1d1fac8f31b45, ce1e7b84cabb446e9cc28003226b6d7e; no voters:
I20260925 04:38:50.129410 1449 raft_consensus.cc:2781] T 60702bbb6cdd400eac661ebf90aea69a P ce1e7b84cabb446e9cc28003226b6d7e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:38:50.129483 1449 raft_consensus.cc:482] T 60702bbb6cdd400eac661ebf90aea69a P ce1e7b84cabb446e9cc28003226b6d7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:38:50.129516 1449 raft_consensus.cc:3037] T 60702bbb6cdd400eac661ebf90aea69a P ce1e7b84cabb446e9cc28003226b6d7e [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:38:50.163558 1449 raft_consensus.cc:504] T 60702bbb6cdd400eac661ebf90aea69a P ce1e7b84cabb446e9cc28003226b6d7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0b1259300b34b318d0094028ee55384" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42947 } } peers { permanent_uuid: "ce1e7b84cabb446e9cc28003226b6d7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37939 } } peers { permanent_uuid: "31cfd1cdbe754020bca1d1fac8f31b45" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40829 } }
I20260925 04:38:50.164108 1449 leader_election.cc:288] T 60702bbb6cdd400eac661ebf90aea69a P ce1e7b84cabb446e9cc28003226b6d7e [CANDIDATE]: Term 1 election: Requested vote from peers c0b1259300b34b318d0094028ee55384 (127.0.98.3:42947), 31cfd1cdbe754020bca1d1fac8f31b45 (127.0.98.1:40829)
I20260925 04:38:50.164660 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60702bbb6cdd400eac661ebf90aea69a" candidate_uuid: "ce1e7b84cabb446e9cc28003226b6d7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31cfd1cdbe754020bca1d1fac8f31b45"
I20260925 04:38:50.164697 1376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60702bbb6cdd400eac661ebf90aea69a" candidate_uuid: "ce1e7b84cabb446e9cc28003226b6d7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0b1259300b34b318d0094028ee55384"
I20260925 04:38:50.164836 1376 raft_consensus.cc:3037] T 60702bbb6cdd400eac661ebf90aea69a P c0b1259300b34b318d0094028ee55384 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:38:50.164836 1107 raft_consensus.cc:3037] T 60702bbb6cdd400eac661ebf90aea69a P 31cfd1cdbe754020bca1d1fac8f31b45 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:38:50.166260 1376 raft_consensus.cc:2454] T 60702bbb6cdd400eac661ebf90aea69a P c0b1259300b34b318d0094028ee55384 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce1e7b84cabb446e9cc28003226b6d7e in term 1.
I20260925 04:38:50.166260 1107 raft_consensus.cc:2454] T 60702bbb6cdd400eac661ebf90aea69a P 31cfd1cdbe754020bca1d1fac8f31b45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce1e7b84cabb446e9cc28003226b6d7e in term 1.
I20260925 04:38:50.166775 1177 leader_election.cc:302] T 60702bbb6cdd400eac661ebf90aea69a P ce1e7b84cabb446e9cc28003226b6d7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 31cfd1cdbe754020bca1d1fac8f31b45, c0b1259300b34b318d0094028ee55384, ce1e7b84cabb446e9cc28003226b6d7e; no voters:
I20260925 04:38:50.167006 1449 raft_consensus.cc:2781] T 60702bbb6cdd400eac661ebf90aea69a P ce1e7b84cabb446e9cc28003226b6d7e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:38:50.167351 1449 raft_consensus.cc:686] T 60702bbb6cdd400eac661ebf90aea69a P ce1e7b84cabb446e9cc28003226b6d7e [term 1 LEADER]: Becoming Leader. State: Replica: ce1e7b84cabb446e9cc28003226b6d7e, State: Running, Role: LEADER
I20260925 04:38:50.167800 1449 consensus_queue.cc:237] T 60702bbb6cdd400eac661ebf90aea69a P ce1e7b84cabb446e9cc28003226b6d7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0b1259300b34b318d0094028ee55384" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42947 } } peers { permanent_uuid: "ce1e7b84cabb446e9cc28003226b6d7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37939 } } peers { permanent_uuid: "31cfd1cdbe754020bca1d1fac8f31b45" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40829 } }
I20260925 04:38:50.171823 962 catalog_manager.cc:5495] T 60702bbb6cdd400eac661ebf90aea69a P ce1e7b84cabb446e9cc28003226b6d7e reported cstate change: term changed from 0 to 1, leader changed from <none> to ce1e7b84cabb446e9cc28003226b6d7e (127.0.98.2). New cstate: current_term: 1 leader_uuid: "ce1e7b84cabb446e9cc28003226b6d7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0b1259300b34b318d0094028ee55384" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce1e7b84cabb446e9cc28003226b6d7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31cfd1cdbe754020bca1d1fac8f31b45" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40829 } health_report { overall_health: UNKNOWN } } }
Sep 25 04:38:50 dist-test-slave-dc95 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 04:38:50.210505 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:50.200836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48272 (local address 127.0.98.1:40829)
0925 04:38:50.201014 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:50.201019 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:50.201180 (+ 161us) server_negotiation.cc:407] Connection header received
0925 04:38:50.201332 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:50.201336 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:50.201381 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:50.201451 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:50.202187 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:50.202770 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:50.203653 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:50.203875 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:50.203973 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:50.206740 (+ 2767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:38:50.206754 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:38:50.206768 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:38:50.206799 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:38:50.208571 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:50.209130 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:50.209136 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:50.209138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:50.209229 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:50.209583 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:50.209585 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:50.209587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:50.209857 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:38:50.210028 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:50.210253 (+ 225us) server_negotiation.cc:299] Negotiation successful
0925 04:38:50.210364 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
Sep 25 04:38:50 dist-test-slave-dc95 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311130, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 04:38:50 dist-test-slave-dc95 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311130, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 04:38:50.257387 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:50.248243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48280 (local address 127.0.98.1:40829)
0925 04:38:50.248445 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:50.248450 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:50.248625 (+ 175us) server_negotiation.cc:407] Connection header received
0925 04:38:50.248753 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:50.248755 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:50.248802 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:50.249005 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:50.249909 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:50.250413 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:50.251397 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:50.251676 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:50.251766 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:50.254108 (+ 2342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:38:50.254112 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:38:50.254115 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:38:50.254143 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:38:50.255548 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:50.256001 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:50.256004 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:50.256005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:50.256083 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:50.256424 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:50.256427 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:50.256428 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:50.256628 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:38:50.256739 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:50.257092 (+ 353us) server_negotiation.cc:299] Negotiation successful
0925 04:38:50.257228 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Sep 25 04:38:50 dist-test-slave-dc95 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 04:38:50 dist-test-slave-dc95 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 04:38:50.338732 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:50.327862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48292 (local address 127.0.98.1:40829)
0925 04:38:50.328022 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:50.328027 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:50.328220 (+ 193us) server_negotiation.cc:407] Connection header received
0925 04:38:50.328352 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:50.328355 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:50.328399 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:50.328464 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:50.329546 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:50.330300 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:50.331277 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:50.331543 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:50.331687 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:50.334328 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:38:50.334331 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:38:50.334334 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:38:50.334367 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:38:50.336330 (+ 1963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:50.337011 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:50.337015 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:50.337017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:50.337130 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:50.337608 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:50.337611 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:50.337613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:50.337803 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:38:50.337916 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:50.338330 (+ 414us) server_negotiation.cc:299] Negotiation successful
0925 04:38:50.338589 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260925 04:38:50.340301 1067 tablet_service.cc:2908] Rejecting scan request for tablet 60702bbb6cdd400eac661ebf90aea69a: Uninitialized: clean time has not yet been initialized
I20260925 04:38:50.342783 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 1024
I20260925 04:38:50.348155 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 1158
I20260925 04:38:50.353302 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 1293
I20260925 04:38:50.358554 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 930
2026-09-25T04:38:50Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4732 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:38:51 dist-test-slave-dc95 krb5kdc[1466](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:38:51 dist-test-slave-dc95 krb5kdc[1466](info): set up 2 sockets
Sep 25 04:38:51 dist-test-slave-dc95 krb5kdc[1466](info): commencing operation
krb5kdc: starting...
W20260925 04:38:52.399021 392 mini_kdc.cc:121] Time spent starting KDC: real 2.018s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260925 04:38:52.615938 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.217s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260925 04:38:52.794538 392 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.178s user 0.000s sys 0.002s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/krb5kdc/test-user.keytab.
Sep 25 04:38:52 dist-test-slave-dc95 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:38:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:38:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 04:38:52.879740 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46757
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46757
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:38:52.994858 1479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:38:52.995167 1479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:38:52.995225 1479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:38:52.999460 1479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:38:52.999548 1479 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 04:38:52.999570 1479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:38:52.999594 1479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:38:52.999641 1479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:38:52.999675 1479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:38:53.005597 1479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46757
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:38:53.006764 1479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:38:53.007858 1479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:38:53.016640 1485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:38:53.016602 1484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:38:53.016659 1479 server_base.cc:1034] running on GCE node
W20260925 04:38:53.016592 1487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:38:53.017540 1479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:38:53.018757 1479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:38:53.019955 1479 hybrid_clock.cc:648] HybridClock initialized: now 1790311133019934 us; error 38 us; skew 500 ppm
Sep 25 04:38:53 dist-test-slave-dc95 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311133, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:38:53.023150 1479 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 04:38:53.024227 1479 webserver.cc:460] Webserver started at http://127.0.98.62:35931/ using document root <none> and password file <none>
I20260925 04:38:53.024796 1479 fs_manager.cc:359] Metadata directory not provided
I20260925 04:38:53.024864 1479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:38:53.025069 1479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:38:53.027211 1479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/master-0/data/instance:
uuid: "118324c32df744868a1da669fdde7718"
format_stamp: "Formatted at 2026-09-25 04:38:53 on dist-test-slave-dc95"
server_key: "1127db06d264acc597d21b7664504e97"
server_key_iv: "c35f2323c13e80aef1529a9f31a2b549"
server_key_version: "encryptionkey@0"
I20260925 04:38:53.027837 1479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/master-0/wal/instance:
uuid: "118324c32df744868a1da669fdde7718"
format_stamp: "Formatted at 2026-09-25 04:38:53 on dist-test-slave-dc95"
server_key: "1127db06d264acc597d21b7664504e97"
server_key_iv: "c35f2323c13e80aef1529a9f31a2b549"
server_key_version: "encryptionkey@0"
I20260925 04:38:53.031925 1479 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:38:53.034796 1494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:38:53.036432 1479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:38:53.036592 1479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/master-0/wal
uuid: "118324c32df744868a1da669fdde7718"
format_stamp: "Formatted at 2026-09-25 04:38:53 on dist-test-slave-dc95"
server_key: "1127db06d264acc597d21b7664504e97"
server_key_iv: "c35f2323c13e80aef1529a9f31a2b549"
server_key_version: "encryptionkey@0"
I20260925 04:38:53.036724 1479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:38:53.065227 1479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:38:53.068643 1479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:38:53.068845 1479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:38:53.076879 1479 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46757
I20260925 04:38:53.076897 1556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46757 every 8 connection(s)
I20260925 04:38:53.077956 1479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
I20260925 04:38:53.081084 1557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:38:53.086911 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 1479
I20260925 04:38:53.087042 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/master-0/wal/instance
I20260925 04:38:53.087100 1557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 118324c32df744868a1da669fdde7718: Bootstrap starting.
I20260925 04:38:53.087291 392 external_mini_cluster.cc:1428] Setting key 3b0df12cf84e86efbdf8315c4e7a64bd
I20260925 04:38:53.090035 1557 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 118324c32df744868a1da669fdde7718: Neither blocks nor log segments found. Creating new log.
I20260925 04:38:53.090902 1557 log.cc:824] T 00000000000000000000000000000000 P 118324c32df744868a1da669fdde7718: Log is configured to *not* fsync() on all Append() calls
I20260925 04:38:53.092970 1557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 118324c32df744868a1da669fdde7718: No bootstrap required, opened a new log
Sep 25 04:38:53 dist-test-slave-dc95 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:38:53.096450 1557 raft_consensus.cc:348] T 00000000000000000000000000000000 P 118324c32df744868a1da669fdde7718 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "118324c32df744868a1da669fdde7718" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46757 } }
I20260925 04:38:53.096714 1557 raft_consensus.cc:374] T 00000000000000000000000000000000 P 118324c32df744868a1da669fdde7718 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:38:53.096771 1557 raft_consensus.cc:729] T 00000000000000000000000000000000 P 118324c32df744868a1da669fdde7718 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 118324c32df744868a1da669fdde7718, State: Initialized, Role: FOLLOWER
I20260925 04:38:53.097240 1557 consensus_queue.cc:260] T 00000000000000000000000000000000 P 118324c32df744868a1da669fdde7718 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "118324c32df744868a1da669fdde7718" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46757 } }
I20260925 04:38:53.097419 1557 raft_consensus.cc:388] T 00000000000000000000000000000000 P 118324c32df744868a1da669fdde7718 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:38:53.097498 1557 raft_consensus.cc:482] T 00000000000000000000000000000000 P 118324c32df744868a1da669fdde7718 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:38:53.097600 1557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 118324c32df744868a1da669fdde7718 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:38:53.098573 1557 raft_consensus.cc:504] T 00000000000000000000000000000000 P 118324c32df744868a1da669fdde7718 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "118324c32df744868a1da669fdde7718" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46757 } }
I20260925 04:38:53.098922 1557 leader_election.cc:302] T 00000000000000000000000000000000 P 118324c32df744868a1da669fdde7718 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 118324c32df744868a1da669fdde7718; no voters:
I20260925 04:38:53.099246 1557 leader_election.cc:288] T 00000000000000000000000000000000 P 118324c32df744868a1da669fdde7718 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:38:53.099385 1562 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 118324c32df744868a1da669fdde7718 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:38:53.099586 1562 raft_consensus.cc:686] T 00000000000000000000000000000000 P 118324c32df744868a1da669fdde7718 [term 1 LEADER]: Becoming Leader. State: Replica: 118324c32df744868a1da669fdde7718, State: Running, Role: LEADER
I20260925 04:38:53.099979 1562 consensus_queue.cc:237] T 00000000000000000000000000000000 P 118324c32df744868a1da669fdde7718 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "118324c32df744868a1da669fdde7718" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46757 } }
I20260925 04:38:53.100687 1557 sys_catalog.cc:564] T 00000000000000000000000000000000 P 118324c32df744868a1da669fdde7718 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:38:53.101585 1563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 118324c32df744868a1da669fdde7718 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "118324c32df744868a1da669fdde7718" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "118324c32df744868a1da669fdde7718" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46757 } } }
I20260925 04:38:53.101720 1563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 118324c32df744868a1da669fdde7718 [sys.catalog]: This master's current role is: LEADER
I20260925 04:38:53.102106 1564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 118324c32df744868a1da669fdde7718 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 118324c32df744868a1da669fdde7718. Latest consensus state: current_term: 1 leader_uuid: "118324c32df744868a1da669fdde7718" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "118324c32df744868a1da669fdde7718" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46757 } } }
I20260925 04:38:53.102144 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:53.088696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37336 (local address 127.0.98.62:46757)
0925 04:38:53.089132 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:53.089148 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:53.089185 (+ 37us) server_negotiation.cc:407] Connection header received
0925 04:38:53.089913 (+ 728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:53.089933 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:53.090347 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:53.090658 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:53.091517 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:53.092566 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:53.093431 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:53.093917 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:53.094487 (+ 570us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:53.096772 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:38:53.096778 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:38:53.096799 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:38:53.096835 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:38:53.098583 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:53.099064 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:53.099071 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:53.099078 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:53.099197 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:53.099541 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:53.099544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:53.099546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:53.100098 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:38:53.100269 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:53.100807 (+ 538us) server_negotiation.cc:299] Negotiation successful
0925 04:38:53.101139 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":139,"threads_started":1}
I20260925 04:38:53.102239 1564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 118324c32df744868a1da669fdde7718 [sys.catalog]: This master's current role is: LEADER
I20260925 04:38:53.103320 1571 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:38:53.106336 1571 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:38:53.111996 1571 catalog_manager.cc:1347] Generated new cluster ID: 543c6eedc022461d9b821691af65e573
I20260925 04:38:53.112082 1571 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:38:53.126405 1571 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:38:53.127492 1571 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:38:53.132256 1571 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 118324c32df744868a1da669fdde7718: Generated new TSK 0
I20260925 04:38:53.133035 1571 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 04:38:53.167156 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46757
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:38:53.280952 1583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:38:53.281248 1583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:38:53.281298 1583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:38:53.285574 1583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:38:53.285648 1583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:38:53.285691 1583 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 04:38:53.285728 1583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 04:38:53.291301 1583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:38:53.292547 1583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:38:53.293613 1583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:38:53.303689 1589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:38:53.303785 1591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:38:53.303858 1583 server_base.cc:1034] running on GCE node
W20260925 04:38:53.303745 1588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:38:53.304348 1583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:38:53.305056 1583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:38:53.306243 1583 hybrid_clock.cc:648] HybridClock initialized: now 1790311133306217 us; error 40 us; skew 500 ppm
Sep 25 04:38:53 dist-test-slave-dc95 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311133, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:38:53.309391 1583 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 04:38:53.310546 1583 webserver.cc:460] Webserver started at http://127.0.98.1:44673/ using document root <none> and password file <none>
I20260925 04:38:53.311224 1583 fs_manager.cc:359] Metadata directory not provided
I20260925 04:38:53.311296 1583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:38:53.311554 1583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:38:53.314031 1583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/ts-0/data/instance:
uuid: "e460d968d94246c18d3569e50e4b6c5e"
format_stamp: "Formatted at 2026-09-25 04:38:53 on dist-test-slave-dc95"
server_key: "04a74a4cae09755564e8ecd73043315e"
server_key_iv: "8826f50a8311da87492a364736bb73c5"
server_key_version: "encryptionkey@0"
I20260925 04:38:53.314690 1583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e460d968d94246c18d3569e50e4b6c5e"
format_stamp: "Formatted at 2026-09-25 04:38:53 on dist-test-slave-dc95"
server_key: "04a74a4cae09755564e8ecd73043315e"
server_key_iv: "8826f50a8311da87492a364736bb73c5"
server_key_version: "encryptionkey@0"
I20260925 04:38:53.319187 1583 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260925 04:38:53.323024 1598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:38:53.324632 1583 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:38:53.324785 1583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/ts-0/wal
uuid: "e460d968d94246c18d3569e50e4b6c5e"
format_stamp: "Formatted at 2026-09-25 04:38:53 on dist-test-slave-dc95"
server_key: "04a74a4cae09755564e8ecd73043315e"
server_key_iv: "8826f50a8311da87492a364736bb73c5"
server_key_version: "encryptionkey@0"
I20260925 04:38:53.324939 1583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:38:53.337713 1583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:38:53.341384 1583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:38:53.341583 1583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:38:53.343174 1583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:38:53.343228 1583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:38:53.343284 1583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:38:53.343312 1583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:38:53.354014 1583 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43563
I20260925 04:38:53.354104 1713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43563 every 8 connection(s)
Sep 25 04:38:53 dist-test-slave-dc95 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311133, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:38:53.355232 1583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
I20260925 04:38:53.359460 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:53.344056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36917 (local address 127.0.98.62:46757)
0925 04:38:53.344172 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:53.344176 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:53.345350 (+ 1174us) server_negotiation.cc:407] Connection header received
0925 04:38:53.346697 (+ 1347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:53.346702 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:53.346767 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:53.346837 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:53.348704 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:53.349520 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:53.351512 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:53.352094 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:53.352251 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:53.355670 (+ 3419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:38:53.355675 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:38:53.355678 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:38:53.355720 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:38:53.357326 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:53.357909 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:53.357912 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:53.357913 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:53.357996 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:53.358333 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:53.358336 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:53.358338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:53.358539 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:38:53.358627 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:53.359146 (+ 519us) server_negotiation.cc:299] Negotiation successful
0925 04:38:53.359278 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 04:38:53.360376 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:53.344670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46757 (local address 127.0.98.1:36917)
0925 04:38:53.345192 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:53.345237 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 04:38:53.346419 (+ 1182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:38:53.347258 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:38:53.347265 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:38:53.347803 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:38:53.348537 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:53.348561 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:53.349690 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:53.349704 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:53.351247 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:53.351261 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:53.352311 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:53.352317 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:53.352636 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:53.353312 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:38:53.353333 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:38:53.355389 (+ 2056us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:38:53.357471 (+ 2082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:53.357478 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:53.357496 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:53.357795 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:53.358085 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:53.358088 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:53.358090 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:53.358238 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:53.358640 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:38:53.358648 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:38:53.358914 (+ 266us) client_negotiation.cc:769] Sending connection context
0925 04:38:53.359091 (+ 177us) client_negotiation.cc:240] Negotiation successful
0925 04:38:53.359477 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":143,"threads_started":1}
I20260925 04:38:53.361701 1714 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46757
I20260925 04:38:53.361891 1714 heartbeater.cc:464] Registering TS with master...
I20260925 04:38:53.362401 1714 heartbeater.cc:511] Master 127.0.98.62:46757 requested a full tablet report, sending...
I20260925 04:38:53.363886 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 1583
I20260925 04:38:53.363994 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/ts-0/wal/instance
I20260925 04:38:53.364233 1511 ts_manager.cc:194] Registered new tserver with Master: e460d968d94246c18d3569e50e4b6c5e (127.0.98.1:43563)
I20260925 04:38:53.364279 392 external_mini_cluster.cc:1428] Setting key 2e8d606684235f7f4ec2c6fd1a691b74
I20260925 04:38:53.365883 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36917
I20260925 04:38:53.371515 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:53.364057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56573 (local address 127.0.98.62:46757)
0925 04:38:53.364166 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:53.364170 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:53.364424 (+ 254us) server_negotiation.cc:407] Connection header received
0925 04:38:53.364531 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:53.364534 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:53.364587 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:53.364671 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:53.365162 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:53.365889 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:53.366922 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:53.367228 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:53.367333 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:53.368529 (+ 1196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:38:53.368532 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:38:53.368535 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:38:53.368560 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:38:53.369852 (+ 1292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:53.370244 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:53.370247 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:53.370249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:53.370322 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:53.370691 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:53.370694 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:53.370696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:53.370892 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:38:53.370989 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:53.371237 (+ 248us) server_negotiation.cc:299] Negotiation successful
0925 04:38:53.371342 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 04:38:53.371526 1720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:53.364020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46757 (local address 127.0.98.1:56573)
0925 04:38:53.364321 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:53.364337 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:38:53.364440 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:38:53.364703 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:38:53.364706 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:38:53.364918 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:38:53.365051 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:53.365059 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:53.366043 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:53.366046 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:53.366818 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:53.366829 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:53.367348 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:53.367352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:53.367461 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:53.368045 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:38:53.368058 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:38:53.368387 (+ 329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:38:53.369959 (+ 1572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:53.369963 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:53.369965 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:53.370158 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:53.370401 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:53.370404 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:53.370406 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:53.370571 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:53.370991 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:38:53.370994 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:38:53.371084 (+ 90us) client_negotiation.cc:769] Sending connection context
0925 04:38:53.371170 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 04:38:53.371327 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":120,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 04:38:53.391350 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46757
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:38:53.514684 1723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:38:53.515005 1723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:38:53.515055 1723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:38:53.519816 1723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:38:53.519910 1723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:38:53.519955 1723 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 04:38:53.519989 1723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 04:38:53.525789 1723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:38:53.526902 1723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:38:53.528038 1723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:38:53.538830 1728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:38:53.538875 1723 server_base.cc:1034] running on GCE node
W20260925 04:38:53.538858 1729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:38:53.538830 1731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:38:53.539512 1723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:38:53.540273 1723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:38:53.541491 1723 hybrid_clock.cc:648] HybridClock initialized: now 1790311133541457 us; error 48 us; skew 500 ppm
Sep 25 04:38:53 dist-test-slave-dc95 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311133, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:38:53.544569 1723 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 04:38:53.545630 1723 webserver.cc:460] Webserver started at http://127.0.98.2:40443/ using document root <none> and password file <none>
I20260925 04:38:53.546221 1723 fs_manager.cc:359] Metadata directory not provided
I20260925 04:38:53.546288 1723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:38:53.546505 1723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:38:53.548969 1723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/ts-1/data/instance:
uuid: "d97dfed0caf14d53b8a09d9b161c1e9a"
format_stamp: "Formatted at 2026-09-25 04:38:53 on dist-test-slave-dc95"
server_key: "f6f756eb5746400bdda57b2230ea3015"
server_key_iv: "4a689246b03a9c1208920a698c10faf9"
server_key_version: "encryptionkey@0"
I20260925 04:38:53.549639 1723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d97dfed0caf14d53b8a09d9b161c1e9a"
format_stamp: "Formatted at 2026-09-25 04:38:53 on dist-test-slave-dc95"
server_key: "f6f756eb5746400bdda57b2230ea3015"
server_key_iv: "4a689246b03a9c1208920a698c10faf9"
server_key_version: "encryptionkey@0"
I20260925 04:38:53.553916 1723 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 04:38:53.557011 1738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:38:53.558558 1723 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:38:53.558720 1723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/ts-1/wal
uuid: "d97dfed0caf14d53b8a09d9b161c1e9a"
format_stamp: "Formatted at 2026-09-25 04:38:53 on dist-test-slave-dc95"
server_key: "f6f756eb5746400bdda57b2230ea3015"
server_key_iv: "4a689246b03a9c1208920a698c10faf9"
server_key_version: "encryptionkey@0"
I20260925 04:38:53.558856 1723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:38:53.585816 1723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:38:53.589480 1723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:38:53.589675 1723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:38:53.591179 1723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:38:53.591231 1723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:38:53.591284 1723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:38:53.591308 1723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 04:38:53 dist-test-slave-dc95 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311133, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:38:53.602954 1723 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32931
I20260925 04:38:53.603077 1853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32931 every 8 connection(s)
I20260925 04:38:53.604027 1723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
I20260925 04:38:53.606112 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:53.592116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34885 (local address 127.0.98.62:46757)
0925 04:38:53.592237 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:53.592241 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:53.593491 (+ 1250us) server_negotiation.cc:407] Connection header received
0925 04:38:53.594639 (+ 1148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:53.594643 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:53.594706 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:53.594785 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:53.596288 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:53.597126 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:53.598764 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:53.599025 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:53.599126 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:53.602019 (+ 2893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:38:53.602023 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:38:53.602026 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:38:53.602057 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:38:53.603887 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:53.604443 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:53.604447 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:53.604449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:53.604550 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:53.604959 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:53.604962 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:53.604964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:53.605186 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:38:53.605270 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:53.605858 (+ 588us) server_negotiation.cc:299] Negotiation successful
0925 04:38:53.605974 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 04:38:53.606971 1747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:53.592755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46757 (local address 127.0.98.2:34885)
0925 04:38:53.593289 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:53.593336 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 04:38:53.594374 (+ 1038us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:38:53.594990 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:38:53.595000 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:38:53.595473 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:38:53.596105 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:53.596127 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:53.597258 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:53.597261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:53.598623 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:53.598637 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:53.599231 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:53.599236 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:53.599407 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:53.600090 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:38:53.600108 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:38:53.601796 (+ 1688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:38:53.604011 (+ 2215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:53.604017 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:53.604033 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:53.604340 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:53.604659 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:53.604662 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:53.604664 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:53.604848 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:53.605283 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:38:53.605289 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:38:53.605593 (+ 304us) client_negotiation.cc:769] Sending connection context
0925 04:38:53.605774 (+ 181us) client_negotiation.cc:240] Negotiation successful
0925 04:38:53.606075 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":145,"threads_started":1}
I20260925 04:38:53.608207 1854 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46757
I20260925 04:38:53.608345 1854 heartbeater.cc:464] Registering TS with master...
I20260925 04:38:53.608901 1854 heartbeater.cc:511] Master 127.0.98.62:46757 requested a full tablet report, sending...
I20260925 04:38:53.608917 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 1723
I20260925 04:38:53.609001 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/ts-1/wal/instance
I20260925 04:38:53.609263 392 external_mini_cluster.cc:1428] Setting key dcdd7cc17d6c6a21f78f51081ac01a3f
I20260925 04:38:53.610157 1511 ts_manager.cc:194] Registered new tserver with Master: d97dfed0caf14d53b8a09d9b161c1e9a (127.0.98.2:32931)
I20260925 04:38:53.610827 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34885
I20260925 04:38:53.618289 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:53.610580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43199 (local address 127.0.98.62:46757)
0925 04:38:53.610696 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:53.610701 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:53.610931 (+ 230us) server_negotiation.cc:407] Connection header received
0925 04:38:53.611101 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:53.611104 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:53.611155 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:53.611234 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:53.611711 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:53.612410 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:53.613650 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:53.613868 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:53.613957 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:53.615298 (+ 1341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:38:53.615302 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:38:53.615303 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:38:53.615328 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:38:53.616724 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:53.617135 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:53.617139 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:53.617140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:53.617219 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:53.617456 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:53.617460 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:53.617462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:53.617688 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:38:53.617782 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:53.618020 (+ 238us) server_negotiation.cc:299] Negotiation successful
0925 04:38:53.618126 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 04:38:53.618299 1860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:53.610511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46757 (local address 127.0.98.2:43199)
0925 04:38:53.610855 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:53.610869 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:38:53.610997 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:38:53.611262 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:38:53.611264 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:38:53.611438 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:38:53.611568 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:53.611574 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:53.612579 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:53.612586 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:53.613537 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:53.613548 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:53.613974 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:53.613977 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:53.614077 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:53.614775 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:38:53.614788 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:38:53.615143 (+ 355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:38:53.616831 (+ 1688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:53.616834 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:53.616836 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:53.617042 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:53.617298 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:53.617301 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:53.617302 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:53.617378 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:53.617776 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:38:53.617779 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:38:53.617868 (+ 89us) client_negotiation.cc:769] Sending connection context
0925 04:38:53.617954 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 04:38:53.618106 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":136,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 04:38:53.639448 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46757
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:38:53.756774 1863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:38:53.757090 1863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:38:53.757138 1863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:38:53.761544 1863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:38:53.761626 1863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:38:53.761672 1863 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 04:38:53.761725 1863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 04:38:53.767581 1863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:38:53.768752 1863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:38:53.769798 1863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:38:53.779945 1871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:38:53.779958 1869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:38:53.780020 1863 server_base.cc:1034] running on GCE node
W20260925 04:38:53.779910 1868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:38:53.780629 1863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:38:53.781296 1863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:38:53.782486 1863 hybrid_clock.cc:648] HybridClock initialized: now 1790311133782456 us; error 46 us; skew 500 ppm
Sep 25 04:38:53 dist-test-slave-dc95 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311133, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:38:53.785413 1863 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 04:38:53.786469 1863 webserver.cc:460] Webserver started at http://127.0.98.3:46515/ using document root <none> and password file <none>
I20260925 04:38:53.787111 1863 fs_manager.cc:359] Metadata directory not provided
I20260925 04:38:53.787184 1863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:38:53.787401 1863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:38:53.789829 1863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/ts-2/data/instance:
uuid: "04a6633bea0742fe8224838b7e84777b"
format_stamp: "Formatted at 2026-09-25 04:38:53 on dist-test-slave-dc95"
server_key: "ab2d2e17b7dfeb9844119a01dbaf8f34"
server_key_iv: "aaefacb6aeeed552042a78b8eb6fc071"
server_key_version: "encryptionkey@0"
I20260925 04:38:53.790490 1863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/ts-2/wal/instance:
uuid: "04a6633bea0742fe8224838b7e84777b"
format_stamp: "Formatted at 2026-09-25 04:38:53 on dist-test-slave-dc95"
server_key: "ab2d2e17b7dfeb9844119a01dbaf8f34"
server_key_iv: "aaefacb6aeeed552042a78b8eb6fc071"
server_key_version: "encryptionkey@0"
I20260925 04:38:53.794860 1863 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 04:38:53.797981 1878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:38:53.799358 1863 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:38:53.799485 1863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/ts-2/wal
uuid: "04a6633bea0742fe8224838b7e84777b"
format_stamp: "Formatted at 2026-09-25 04:38:53 on dist-test-slave-dc95"
server_key: "ab2d2e17b7dfeb9844119a01dbaf8f34"
server_key_iv: "aaefacb6aeeed552042a78b8eb6fc071"
server_key_version: "encryptionkey@0"
I20260925 04:38:53.799638 1863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:38:53.816394 1863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:38:53.819643 1863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:38:53.819828 1863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:38:53.821309 1863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:38:53.821363 1863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:38:53.821412 1863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:38:53.821439 1863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 04:38:53 dist-test-slave-dc95 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311133, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:38:53.831530 1863 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34353
I20260925 04:38:53.831586 1993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34353 every 8 connection(s)
I20260925 04:38:53.832511 1863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
I20260925 04:38:53.834954 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:53.822088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57603 (local address 127.0.98.62:46757)
0925 04:38:53.822224 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:53.822229 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:53.823195 (+ 966us) server_negotiation.cc:407] Connection header received
0925 04:38:53.824294 (+ 1099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:53.824297 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:53.824357 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:53.824427 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:53.825723 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:53.826430 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:53.827772 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:53.828020 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:53.828148 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:53.831073 (+ 2925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:38:53.831077 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:38:53.831080 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:38:53.831109 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:38:53.832637 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:53.833237 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:53.833241 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:53.833243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:53.833346 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:53.833700 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:53.833703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:53.833704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:53.833974 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:38:53.834061 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:53.834639 (+ 578us) server_negotiation.cc:299] Negotiation successful
0925 04:38:53.834771 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 04:38:53.835834 1887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:53.822556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46757 (local address 127.0.98.3:57603)
0925 04:38:53.823013 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:53.823065 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 04:38:53.824064 (+ 999us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:38:53.824621 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:38:53.824628 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:38:53.824977 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:38:53.825573 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:53.825591 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:53.826603 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:53.826608 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:53.827655 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:53.827668 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:53.828222 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:53.828226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:53.828390 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:53.829036 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:38:53.829070 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:38:53.830872 (+ 1802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:38:53.832750 (+ 1878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:53.832756 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:53.832772 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:53.833109 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:53.833445 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:53.833448 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:53.833450 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:53.833607 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:53.834102 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:38:53.834108 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:38:53.834364 (+ 256us) client_negotiation.cc:769] Sending connection context
0925 04:38:53.834556 (+ 192us) client_negotiation.cc:240] Negotiation successful
0925 04:38:53.834843 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":109,"threads_started":1}
I20260925 04:38:53.836889 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 1863
I20260925 04:38:53.837004 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/ts-2/wal/instance
I20260925 04:38:53.837282 392 external_mini_cluster.cc:1428] Setting key 8107043d9df5c1b26e3bb02bf185a51e
I20260925 04:38:53.837365 1994 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46757
I20260925 04:38:53.837512 1994 heartbeater.cc:464] Registering TS with master...
I20260925 04:38:53.838063 1994 heartbeater.cc:511] Master 127.0.98.62:46757 requested a full tablet report, sending...
I20260925 04:38:53.839164 1510 ts_manager.cc:194] Registered new tserver with Master: 04a6633bea0742fe8224838b7e84777b (127.0.98.3:34353)
I20260925 04:38:53.839895 1510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57603
I20260925 04:38:53.841662 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:38:53.846905 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:53.839395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53371 (local address 127.0.98.62:46757)
0925 04:38:53.839501 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:53.839505 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:53.839819 (+ 314us) server_negotiation.cc:407] Connection header received
0925 04:38:53.839942 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:53.839945 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:53.839993 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:53.840069 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:53.840577 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:53.841290 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:53.842480 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:53.842696 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:53.842774 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:53.843971 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:38:53.843975 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:38:53.843978 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:38:53.844005 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:38:53.845409 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:53.845817 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:53.845821 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:53.845823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:53.845896 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:53.846140 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:53.846144 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:53.846146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:53.846323 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:38:53.846407 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:53.846644 (+ 237us) server_negotiation.cc:299] Negotiation successful
0925 04:38:53.846747 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 04:38:53.846915 2000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:53.839425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46757 (local address 127.0.98.3:53371)
0925 04:38:53.839745 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:53.839757 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:38:53.839847 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:38:53.840089 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:38:53.840092 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:38:53.840309 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:38:53.840438 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:53.840446 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:53.841419 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:53.841422 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:53.842346 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:53.842357 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:53.842800 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:53.842804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:53.842892 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:53.843374 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:38:53.843387 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:38:53.843829 (+ 442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:38:53.845534 (+ 1705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:53.845537 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:53.845540 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:53.845732 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:53.845982 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:53.845985 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:53.845987 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:53.846063 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:53.846406 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:38:53.846409 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:38:53.846497 (+ 88us) client_negotiation.cc:769] Sending connection context
0925 04:38:53.846581 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 04:38:53.846729 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":123,"threads_started":1}
Sep 25 04:38:53 dist-test-slave-dc95 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311133, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 04:38:53 dist-test-slave-dc95 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311133, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:38:53.869779 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:53.859975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37338 (local address 127.0.98.62:46757)
0925 04:38:53.860131 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:53.860136 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:53.860149 (+ 13us) server_negotiation.cc:407] Connection header received
0925 04:38:53.860195 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:53.860199 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:53.860242 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:53.860313 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:53.861271 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:53.862132 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:53.863247 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:53.863574 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:53.863773 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:53.866550 (+ 2777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:38:53.866554 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:38:53.866556 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:38:53.866588 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:38:53.868015 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:53.868481 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:53.868484 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:53.868486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:53.868576 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:53.868984 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:53.868987 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:53.868989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:53.869196 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:38:53.869316 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:53.869509 (+ 193us) server_negotiation.cc:299] Negotiation successful
0925 04:38:53.869612 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 04:38:53.872959 1511 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37338:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:38:53.876048 1511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:38:53.891644 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:53.885894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41612 (local address 127.0.98.1:43563)
0925 04:38:53.886171 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:53.886178 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:53.886618 (+ 440us) server_negotiation.cc:407] Connection header received
0925 04:38:53.886697 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:53.886703 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:53.886920 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:53.887052 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:38:53.887410 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:53.888171 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:53.890293 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:53.890875 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:53.890977 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:53.891145 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:53.891332 (+ 187us) server_negotiation.cc:299] Negotiation successful
0925 04:38:53.891447 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":92,"threads_started":1}
I20260925 04:38:53.891685 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:53.885047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32931 (local address 127.0.0.1:39310)
0925 04:38:53.885455 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:53.885512 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 04:38:53.885650 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:38:53.886518 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:38:53.886524 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:38:53.886544 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:38:53.886701 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:53.886707 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:53.888344 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:53.888347 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:53.889931 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:53.889944 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:53.891106 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:53.891110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:53.891197 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:53.891220 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 04:38:53.891270 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 04:38:53.891471 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":182,"thread_start_us":84,"threads_started":1}
I20260925 04:38:53.891710 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:53.885067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43563 (local address 127.0.0.1:41612)
0925 04:38:53.886406 (+ 1339us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:53.886419 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:38:53.886528 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:38:53.887181 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:38:53.887184 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:38:53.887200 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:38:53.887291 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:53.887297 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:53.888281 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:53.888285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:53.890177 (+ 1892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:53.890186 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:53.890992 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:53.890996 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:53.891132 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:53.891177 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 04:38:53.891289 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 04:38:53.891469 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1275,"mutex_wait_us":353,"spinlock_wait_cycles":13568,"thread_start_us":101,"threads_started":1}
I20260925 04:38:53.891992 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:53.885987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37486 (local address 127.0.98.3:34353)
0925 04:38:53.886415 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:53.886426 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:53.886644 (+ 218us) server_negotiation.cc:407] Connection header received
0925 04:38:53.886763 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:53.886769 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:53.886966 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:53.887112 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:38:53.887453 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:53.888641 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:53.890199 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:53.890969 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:53.891074 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:53.891229 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:53.891748 (+ 519us) server_negotiation.cc:299] Negotiation successful
0925 04:38:53.891854 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":112,"threads_started":1}
I20260925 04:38:53.891685 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:53.885755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34353 (local address 127.0.0.1:37486)
0925 04:38:53.886544 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:53.886557 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:38:53.886659 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:38:53.887246 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:38:53.887249 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:38:53.887265 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:38:53.887342 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:53.887348 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:53.888781 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:53.888784 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:53.890064 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:53.890073 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:53.891281 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:53.891284 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:53.891357 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:53.891376 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 04:38:53.891443 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 04:38:53.891499 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":733,"mutex_wait_us":182,"thread_start_us":92,"threads_started":1}
I20260925 04:38:53.892905 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:53.885166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39310 (local address 127.0.98.2:32931)
0925 04:38:53.885877 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:53.885889 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:53.885913 (+ 24us) server_negotiation.cc:407] Connection header received
0925 04:38:53.886039 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:53.886048 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:53.886311 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:53.886854 (+ 543us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:38:53.887014 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:53.888206 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:53.890078 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:53.890969 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:53.892444 (+ 1475us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:53.892617 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:53.892664 (+ 47us) server_negotiation.cc:299] Negotiation successful
0925 04:38:53.892758 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":109,"threads_started":1}
I20260925 04:38:53.895174 1788 tablet_service.cc:1467] Processing CreateTablet for tablet 9ba1e12fb74b46a4a351bcbb97a6f71f (DEFAULT_TABLE table=test-table [id=e1c59c908c1a41778eb4f5d395ca9206]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:38:53.895215 1928 tablet_service.cc:1467] Processing CreateTablet for tablet 9ba1e12fb74b46a4a351bcbb97a6f71f (DEFAULT_TABLE table=test-table [id=e1c59c908c1a41778eb4f5d395ca9206]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:38:53.895202 1648 tablet_service.cc:1467] Processing CreateTablet for tablet 9ba1e12fb74b46a4a351bcbb97a6f71f (DEFAULT_TABLE table=test-table [id=e1c59c908c1a41778eb4f5d395ca9206]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:38:53.896222 1788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ba1e12fb74b46a4a351bcbb97a6f71f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:38:53.896260 1928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ba1e12fb74b46a4a351bcbb97a6f71f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:38:53.896255 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ba1e12fb74b46a4a351bcbb97a6f71f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:38:53.904466 2018 tablet_bootstrap.cc:492] T 9ba1e12fb74b46a4a351bcbb97a6f71f P e460d968d94246c18d3569e50e4b6c5e: Bootstrap starting.
I20260925 04:38:53.906469 2018 tablet_bootstrap.cc:654] T 9ba1e12fb74b46a4a351bcbb97a6f71f P e460d968d94246c18d3569e50e4b6c5e: Neither blocks nor log segments found. Creating new log.
I20260925 04:38:53.907186 2018 log.cc:824] T 9ba1e12fb74b46a4a351bcbb97a6f71f P e460d968d94246c18d3569e50e4b6c5e: Log is configured to *not* fsync() on all Append() calls
I20260925 04:38:53.907719 2017 tablet_bootstrap.cc:492] T 9ba1e12fb74b46a4a351bcbb97a6f71f P d97dfed0caf14d53b8a09d9b161c1e9a: Bootstrap starting.
I20260925 04:38:53.907874 2019 tablet_bootstrap.cc:492] T 9ba1e12fb74b46a4a351bcbb97a6f71f P 04a6633bea0742fe8224838b7e84777b: Bootstrap starting.
I20260925 04:38:53.909680 2018 tablet_bootstrap.cc:492] T 9ba1e12fb74b46a4a351bcbb97a6f71f P e460d968d94246c18d3569e50e4b6c5e: No bootstrap required, opened a new log
I20260925 04:38:53.909897 2018 ts_tablet_manager.cc:1397] T 9ba1e12fb74b46a4a351bcbb97a6f71f P e460d968d94246c18d3569e50e4b6c5e: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260925 04:38:53.910130 2019 tablet_bootstrap.cc:654] T 9ba1e12fb74b46a4a351bcbb97a6f71f P 04a6633bea0742fe8224838b7e84777b: Neither blocks nor log segments found. Creating new log.
I20260925 04:38:53.910130 2017 tablet_bootstrap.cc:654] T 9ba1e12fb74b46a4a351bcbb97a6f71f P d97dfed0caf14d53b8a09d9b161c1e9a: Neither blocks nor log segments found. Creating new log.
I20260925 04:38:53.911002 2019 log.cc:824] T 9ba1e12fb74b46a4a351bcbb97a6f71f P 04a6633bea0742fe8224838b7e84777b: Log is configured to *not* fsync() on all Append() calls
I20260925 04:38:53.911002 2017 log.cc:824] T 9ba1e12fb74b46a4a351bcbb97a6f71f P d97dfed0caf14d53b8a09d9b161c1e9a: Log is configured to *not* fsync() on all Append() calls
I20260925 04:38:53.913121 2017 tablet_bootstrap.cc:492] T 9ba1e12fb74b46a4a351bcbb97a6f71f P d97dfed0caf14d53b8a09d9b161c1e9a: No bootstrap required, opened a new log
I20260925 04:38:53.913339 2017 ts_tablet_manager.cc:1397] T 9ba1e12fb74b46a4a351bcbb97a6f71f P d97dfed0caf14d53b8a09d9b161c1e9a: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.004s
I20260925 04:38:53.913391 2019 tablet_bootstrap.cc:492] T 9ba1e12fb74b46a4a351bcbb97a6f71f P 04a6633bea0742fe8224838b7e84777b: No bootstrap required, opened a new log
I20260925 04:38:53.913607 2019 ts_tablet_manager.cc:1397] T 9ba1e12fb74b46a4a351bcbb97a6f71f P 04a6633bea0742fe8224838b7e84777b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 04:38:53.914057 2018 raft_consensus.cc:348] T 9ba1e12fb74b46a4a351bcbb97a6f71f P e460d968d94246c18d3569e50e4b6c5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d97dfed0caf14d53b8a09d9b161c1e9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32931 } } peers { permanent_uuid: "e460d968d94246c18d3569e50e4b6c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43563 } } peers { permanent_uuid: "04a6633bea0742fe8224838b7e84777b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34353 } }
I20260925 04:38:53.914371 2018 raft_consensus.cc:374] T 9ba1e12fb74b46a4a351bcbb97a6f71f P e460d968d94246c18d3569e50e4b6c5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:38:53.914433 2018 raft_consensus.cc:729] T 9ba1e12fb74b46a4a351bcbb97a6f71f P e460d968d94246c18d3569e50e4b6c5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e460d968d94246c18d3569e50e4b6c5e, State: Initialized, Role: FOLLOWER
I20260925 04:38:53.914958 2018 consensus_queue.cc:260] T 9ba1e12fb74b46a4a351bcbb97a6f71f P e460d968d94246c18d3569e50e4b6c5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d97dfed0caf14d53b8a09d9b161c1e9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32931 } } peers { permanent_uuid: "e460d968d94246c18d3569e50e4b6c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43563 } } peers { permanent_uuid: "04a6633bea0742fe8224838b7e84777b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34353 } }
I20260925 04:38:53.916177 2018 ts_tablet_manager.cc:1428] T 9ba1e12fb74b46a4a351bcbb97a6f71f P e460d968d94246c18d3569e50e4b6c5e: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 04:38:53.916558 1714 heartbeater.cc:503] Master 127.0.98.62:46757 was elected leader, sending a full tablet report...
I20260925 04:38:53.916970 2017 raft_consensus.cc:348] T 9ba1e12fb74b46a4a351bcbb97a6f71f P d97dfed0caf14d53b8a09d9b161c1e9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d97dfed0caf14d53b8a09d9b161c1e9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32931 } } peers { permanent_uuid: "e460d968d94246c18d3569e50e4b6c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43563 } } peers { permanent_uuid: "04a6633bea0742fe8224838b7e84777b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34353 } }
I20260925 04:38:53.917210 2017 raft_consensus.cc:374] T 9ba1e12fb74b46a4a351bcbb97a6f71f P d97dfed0caf14d53b8a09d9b161c1e9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:38:53.917263 2017 raft_consensus.cc:729] T 9ba1e12fb74b46a4a351bcbb97a6f71f P d97dfed0caf14d53b8a09d9b161c1e9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d97dfed0caf14d53b8a09d9b161c1e9a, State: Initialized, Role: FOLLOWER
I20260925 04:38:53.917449 2019 raft_consensus.cc:348] T 9ba1e12fb74b46a4a351bcbb97a6f71f P 04a6633bea0742fe8224838b7e84777b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d97dfed0caf14d53b8a09d9b161c1e9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32931 } } peers { permanent_uuid: "e460d968d94246c18d3569e50e4b6c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43563 } } peers { permanent_uuid: "04a6633bea0742fe8224838b7e84777b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34353 } }
I20260925 04:38:53.917755 2019 raft_consensus.cc:374] T 9ba1e12fb74b46a4a351bcbb97a6f71f P 04a6633bea0742fe8224838b7e84777b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:38:53.917814 2019 raft_consensus.cc:729] T 9ba1e12fb74b46a4a351bcbb97a6f71f P 04a6633bea0742fe8224838b7e84777b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04a6633bea0742fe8224838b7e84777b, State: Initialized, Role: FOLLOWER
I20260925 04:38:53.917726 2017 consensus_queue.cc:260] T 9ba1e12fb74b46a4a351bcbb97a6f71f P d97dfed0caf14d53b8a09d9b161c1e9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d97dfed0caf14d53b8a09d9b161c1e9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32931 } } peers { permanent_uuid: "e460d968d94246c18d3569e50e4b6c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43563 } } peers { permanent_uuid: "04a6633bea0742fe8224838b7e84777b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34353 } }
I20260925 04:38:53.918234 2019 consensus_queue.cc:260] T 9ba1e12fb74b46a4a351bcbb97a6f71f P 04a6633bea0742fe8224838b7e84777b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d97dfed0caf14d53b8a09d9b161c1e9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32931 } } peers { permanent_uuid: "e460d968d94246c18d3569e50e4b6c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43563 } } peers { permanent_uuid: "04a6633bea0742fe8224838b7e84777b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34353 } }
I20260925 04:38:53.918723 2017 ts_tablet_manager.cc:1428] T 9ba1e12fb74b46a4a351bcbb97a6f71f P d97dfed0caf14d53b8a09d9b161c1e9a: Time spent starting tablet: real 0.005s user 0.000s sys 0.006s
I20260925 04:38:53.918967 1994 heartbeater.cc:503] Master 127.0.98.62:46757 was elected leader, sending a full tablet report...
I20260925 04:38:53.919021 1854 heartbeater.cc:503] Master 127.0.98.62:46757 was elected leader, sending a full tablet report...
I20260925 04:38:53.919217 2019 ts_tablet_manager.cc:1428] T 9ba1e12fb74b46a4a351bcbb97a6f71f P 04a6633bea0742fe8224838b7e84777b: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 04:38:53.983584 2025 raft_consensus.cc:482] T 9ba1e12fb74b46a4a351bcbb97a6f71f P 04a6633bea0742fe8224838b7e84777b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:38:53.983851 2025 raft_consensus.cc:504] T 9ba1e12fb74b46a4a351bcbb97a6f71f P 04a6633bea0742fe8224838b7e84777b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d97dfed0caf14d53b8a09d9b161c1e9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32931 } } peers { permanent_uuid: "e460d968d94246c18d3569e50e4b6c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43563 } } peers { permanent_uuid: "04a6633bea0742fe8224838b7e84777b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34353 } }
I20260925 04:38:53.985042 2025 leader_election.cc:288] T 9ba1e12fb74b46a4a351bcbb97a6f71f P 04a6633bea0742fe8224838b7e84777b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d97dfed0caf14d53b8a09d9b161c1e9a (127.0.98.2:32931), e460d968d94246c18d3569e50e4b6c5e (127.0.98.1:43563)
I20260925 04:38:53.989708 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:53.985396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42253 (local address 127.0.98.2:32931)
0925 04:38:53.985558 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:53.985567 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:53.985582 (+ 15us) server_negotiation.cc:407] Connection header received
0925 04:38:53.985636 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:53.985640 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:53.985690 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:53.985769 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:38:53.986069 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:53.986868 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:53.988530 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:53.989127 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:53.989241 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:53.989311 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:53.989503 (+ 192us) server_negotiation.cc:299] Negotiation successful
0925 04:38:53.989581 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 04:38:53.989710 1887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:53.985261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32931 (local address 127.0.98.3:42253)
0925 04:38:53.985380 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:53.985395 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:38:53.985484 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:38:53.985808 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:38:53.985811 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:38:53.985848 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:38:53.985969 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:53.985975 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:53.987050 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:53.987054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:53.988373 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:53.988385 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:53.989270 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:53.989273 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:53.989357 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:53.989378 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 04:38:53.989474 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 04:38:53.989558 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"spinlock_wait_cycles":25088}
I20260925 04:38:53.990078 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:53.985575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46189 (local address 127.0.98.1:43563)
0925 04:38:53.985692 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:53.985696 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:53.985827 (+ 131us) server_negotiation.cc:407] Connection header received
0925 04:38:53.985941 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:53.985943 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:53.985983 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:53.986050 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:38:53.986286 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:53.986986 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:53.988882 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:53.989477 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:53.989579 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:53.989635 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:53.989850 (+ 215us) server_negotiation.cc:299] Negotiation successful
0925 04:38:53.989962 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 04:38:53.990423 2026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:53.985307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43563 (local address 127.0.98.3:46189)
0925 04:38:53.985734 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:53.985745 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 04:38:53.985854 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:38:53.986079 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:38:53.986082 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:38:53.986099 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:38:53.986189 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:53.986193 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:53.987095 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:53.987098 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:53.988725 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:53.988736 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:53.989615 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:53.989618 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:53.989704 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:53.989726 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 04:38:53.990215 (+ 489us) client_negotiation.cc:240] Negotiation successful
0925 04:38:53.990289 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"spinlock_wait_cycles":6016,"thread_start_us":157,"threads_started":1}
I20260925 04:38:53.990434 1808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ba1e12fb74b46a4a351bcbb97a6f71f" candidate_uuid: "04a6633bea0742fe8224838b7e84777b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d97dfed0caf14d53b8a09d9b161c1e9a" is_pre_election: true
I20260925 04:38:53.990748 1808 raft_consensus.cc:2454] T 9ba1e12fb74b46a4a351bcbb97a6f71f P d97dfed0caf14d53b8a09d9b161c1e9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04a6633bea0742fe8224838b7e84777b in term 0.
I20260925 04:38:53.991037 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ba1e12fb74b46a4a351bcbb97a6f71f" candidate_uuid: "04a6633bea0742fe8224838b7e84777b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e460d968d94246c18d3569e50e4b6c5e" is_pre_election: true
I20260925 04:38:53.991266 1882 leader_election.cc:302] T 9ba1e12fb74b46a4a351bcbb97a6f71f P 04a6633bea0742fe8224838b7e84777b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04a6633bea0742fe8224838b7e84777b, d97dfed0caf14d53b8a09d9b161c1e9a; no voters:
I20260925 04:38:53.991344 1668 raft_consensus.cc:2454] T 9ba1e12fb74b46a4a351bcbb97a6f71f P e460d968d94246c18d3569e50e4b6c5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04a6633bea0742fe8224838b7e84777b in term 0.
I20260925 04:38:53.991575 2025 raft_consensus.cc:2781] T 9ba1e12fb74b46a4a351bcbb97a6f71f P 04a6633bea0742fe8224838b7e84777b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:38:53.991710 2025 raft_consensus.cc:482] T 9ba1e12fb74b46a4a351bcbb97a6f71f P 04a6633bea0742fe8224838b7e84777b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:38:53.991770 2025 raft_consensus.cc:3037] T 9ba1e12fb74b46a4a351bcbb97a6f71f P 04a6633bea0742fe8224838b7e84777b [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:38:53.992808 2025 raft_consensus.cc:504] T 9ba1e12fb74b46a4a351bcbb97a6f71f P 04a6633bea0742fe8224838b7e84777b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d97dfed0caf14d53b8a09d9b161c1e9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32931 } } peers { permanent_uuid: "e460d968d94246c18d3569e50e4b6c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43563 } } peers { permanent_uuid: "04a6633bea0742fe8224838b7e84777b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34353 } }
I20260925 04:38:53.993165 2025 leader_election.cc:288] T 9ba1e12fb74b46a4a351bcbb97a6f71f P 04a6633bea0742fe8224838b7e84777b [CANDIDATE]: Term 1 election: Requested vote from peers d97dfed0caf14d53b8a09d9b161c1e9a (127.0.98.2:32931), e460d968d94246c18d3569e50e4b6c5e (127.0.98.1:43563)
I20260925 04:38:53.993556 1808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ba1e12fb74b46a4a351bcbb97a6f71f" candidate_uuid: "04a6633bea0742fe8224838b7e84777b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d97dfed0caf14d53b8a09d9b161c1e9a"
I20260925 04:38:53.993600 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ba1e12fb74b46a4a351bcbb97a6f71f" candidate_uuid: "04a6633bea0742fe8224838b7e84777b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e460d968d94246c18d3569e50e4b6c5e"
I20260925 04:38:53.993686 1808 raft_consensus.cc:3037] T 9ba1e12fb74b46a4a351bcbb97a6f71f P d97dfed0caf14d53b8a09d9b161c1e9a [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:38:53.993726 1668 raft_consensus.cc:3037] T 9ba1e12fb74b46a4a351bcbb97a6f71f P e460d968d94246c18d3569e50e4b6c5e [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:38:53.994867 1668 raft_consensus.cc:2454] T 9ba1e12fb74b46a4a351bcbb97a6f71f P e460d968d94246c18d3569e50e4b6c5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04a6633bea0742fe8224838b7e84777b in term 1.
I20260925 04:38:53.995087 1808 raft_consensus.cc:2454] T 9ba1e12fb74b46a4a351bcbb97a6f71f P d97dfed0caf14d53b8a09d9b161c1e9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04a6633bea0742fe8224838b7e84777b in term 1.
I20260925 04:38:53.995237 1880 leader_election.cc:302] T 9ba1e12fb74b46a4a351bcbb97a6f71f P 04a6633bea0742fe8224838b7e84777b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04a6633bea0742fe8224838b7e84777b, e460d968d94246c18d3569e50e4b6c5e; no voters:
I20260925 04:38:53.995438 2025 raft_consensus.cc:2781] T 9ba1e12fb74b46a4a351bcbb97a6f71f P 04a6633bea0742fe8224838b7e84777b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:38:53.995710 2025 raft_consensus.cc:686] T 9ba1e12fb74b46a4a351bcbb97a6f71f P 04a6633bea0742fe8224838b7e84777b [term 1 LEADER]: Becoming Leader. State: Replica: 04a6633bea0742fe8224838b7e84777b, State: Running, Role: LEADER
I20260925 04:38:53.996086 2025 consensus_queue.cc:237] T 9ba1e12fb74b46a4a351bcbb97a6f71f P 04a6633bea0742fe8224838b7e84777b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d97dfed0caf14d53b8a09d9b161c1e9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32931 } } peers { permanent_uuid: "e460d968d94246c18d3569e50e4b6c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43563 } } peers { permanent_uuid: "04a6633bea0742fe8224838b7e84777b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34353 } }
I20260925 04:38:53.999393 1510 catalog_manager.cc:5495] T 9ba1e12fb74b46a4a351bcbb97a6f71f P 04a6633bea0742fe8224838b7e84777b reported cstate change: term changed from 0 to 1, leader changed from <none> to 04a6633bea0742fe8224838b7e84777b (127.0.98.3). New cstate: current_term: 1 leader_uuid: "04a6633bea0742fe8224838b7e84777b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d97dfed0caf14d53b8a09d9b161c1e9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e460d968d94246c18d3569e50e4b6c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04a6633bea0742fe8224838b7e84777b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34353 } health_report { overall_health: HEALTHY } } }
I20260925 04:38:54.021621 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:54.017801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37500 (local address 127.0.98.3:34353)
0925 04:38:54.017914 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:54.017917 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:54.017936 (+ 19us) server_negotiation.cc:407] Connection header received
0925 04:38:54.018197 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:54.018200 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:54.018257 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:54.018346 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:38:54.018623 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:54.019116 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:54.020152 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:54.020387 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:54.020478 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:54.020774 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:38:54.021262 (+ 488us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:38:54.021355 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:54.021425 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 04:38:54.021483 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 04:38:54.027967 1668 raft_consensus.cc:1261] T 9ba1e12fb74b46a4a351bcbb97a6f71f P e460d968d94246c18d3569e50e4b6c5e [term 1 FOLLOWER]: Refusing update from remote peer 04a6633bea0742fe8224838b7e84777b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:38:54.027967 1808 raft_consensus.cc:1261] T 9ba1e12fb74b46a4a351bcbb97a6f71f P d97dfed0caf14d53b8a09d9b161c1e9a [term 1 FOLLOWER]: Refusing update from remote peer 04a6633bea0742fe8224838b7e84777b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:38:54.028707 2025 consensus_queue.cc:1035] T 9ba1e12fb74b46a4a351bcbb97a6f71f P 04a6633bea0742fe8224838b7e84777b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d97dfed0caf14d53b8a09d9b161c1e9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:38:54.028909 2027 consensus_queue.cc:1035] T 9ba1e12fb74b46a4a351bcbb97a6f71f P 04a6633bea0742fe8224838b7e84777b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e460d968d94246c18d3569e50e4b6c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:38:54.039153 2031 mvcc.cc:205] Tried to move back new op lower bound from 7333114404970143744 to 7333114404850704384. Current Snapshot: MvccSnapshot[applied={T|T < 7333114404970143744}]
I20260925 04:38:54.049436 1510 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37338:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:38:54.049672 1510 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37338:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:38:54.052567 1510 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 118324c32df744868a1da669fdde7718: Sending DeleteTablet for 3 replicas of tablet 9ba1e12fb74b46a4a351bcbb97a6f71f
I20260925 04:38:54.053464 1788 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ba1e12fb74b46a4a351bcbb97a6f71f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:38:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39310
I20260925 04:38:54.053520 1648 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ba1e12fb74b46a4a351bcbb97a6f71f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:38:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41612
I20260925 04:38:54.054090 2041 tablet_replica.cc:330] T 9ba1e12fb74b46a4a351bcbb97a6f71f P e460d968d94246c18d3569e50e4b6c5e: stopping tablet replica
I20260925 04:38:54.054060 1928 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ba1e12fb74b46a4a351bcbb97a6f71f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:38:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37486
I20260925 04:38:54.054422 2041 raft_consensus.cc:2229] T 9ba1e12fb74b46a4a351bcbb97a6f71f P e460d968d94246c18d3569e50e4b6c5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:38:54.054708 2042 tablet_replica.cc:330] T 9ba1e12fb74b46a4a351bcbb97a6f71f P 04a6633bea0742fe8224838b7e84777b: stopping tablet replica
I20260925 04:38:54.054785 2041 raft_consensus.cc:2258] T 9ba1e12fb74b46a4a351bcbb97a6f71f P e460d968d94246c18d3569e50e4b6c5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:38:54.054697 1510 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37338:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:38:54.055135 1510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:38:54.055176 2040 tablet_replica.cc:330] T 9ba1e12fb74b46a4a351bcbb97a6f71f P d97dfed0caf14d53b8a09d9b161c1e9a: stopping tablet replica
I20260925 04:38:54.055471 2040 raft_consensus.cc:2229] T 9ba1e12fb74b46a4a351bcbb97a6f71f P d97dfed0caf14d53b8a09d9b161c1e9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:38:54.055812 2040 raft_consensus.cc:2258] T 9ba1e12fb74b46a4a351bcbb97a6f71f P d97dfed0caf14d53b8a09d9b161c1e9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:38:54.056798 2041 ts_tablet_manager.cc:1905] T 9ba1e12fb74b46a4a351bcbb97a6f71f P e460d968d94246c18d3569e50e4b6c5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:38:54.057345 2040 ts_tablet_manager.cc:1905] T 9ba1e12fb74b46a4a351bcbb97a6f71f P d97dfed0caf14d53b8a09d9b161c1e9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:38:54.059677 2041 ts_tablet_manager.cc:1918] T 9ba1e12fb74b46a4a351bcbb97a6f71f P e460d968d94246c18d3569e50e4b6c5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:38:54.059948 2040 ts_tablet_manager.cc:1918] T 9ba1e12fb74b46a4a351bcbb97a6f71f P d97dfed0caf14d53b8a09d9b161c1e9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:38:54.060056 2040 log.cc:1196] T 9ba1e12fb74b46a4a351bcbb97a6f71f P d97dfed0caf14d53b8a09d9b161c1e9a: Deleting WAL directory at /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/ts-1/wal/wals/9ba1e12fb74b46a4a351bcbb97a6f71f
I20260925 04:38:54.060141 2042 raft_consensus.cc:2229] T 9ba1e12fb74b46a4a351bcbb97a6f71f P 04a6633bea0742fe8224838b7e84777b [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:38:54.060354 2040 ts_tablet_manager.cc:1939] T 9ba1e12fb74b46a4a351bcbb97a6f71f P d97dfed0caf14d53b8a09d9b161c1e9a: Deleting consensus metadata
I20260925 04:38:54.060604 2042 raft_consensus.cc:2258] T 9ba1e12fb74b46a4a351bcbb97a6f71f P 04a6633bea0742fe8224838b7e84777b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:38:54.060747 2041 log.cc:1196] T 9ba1e12fb74b46a4a351bcbb97a6f71f P e460d968d94246c18d3569e50e4b6c5e: Deleting WAL directory at /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/ts-0/wal/wals/9ba1e12fb74b46a4a351bcbb97a6f71f
I20260925 04:38:54.060957 1788 tablet_service.cc:1467] Processing CreateTablet for tablet e8b33cdd11ba431cbcde2d9ca97700df (DEFAULT_TABLE table=test-table [id=38c427f7766e4477949ea8c49318d744]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:38:54.061203 1788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8b33cdd11ba431cbcde2d9ca97700df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:38:54.061339 2041 ts_tablet_manager.cc:1939] T 9ba1e12fb74b46a4a351bcbb97a6f71f P e460d968d94246c18d3569e50e4b6c5e: Deleting consensus metadata
I20260925 04:38:54.061800 1498 catalog_manager.cc:4841] TS d97dfed0caf14d53b8a09d9b161c1e9a (127.0.98.2:32931): tablet 9ba1e12fb74b46a4a351bcbb97a6f71f (table test-table [id=e1c59c908c1a41778eb4f5d395ca9206]) successfully deleted
I20260925 04:38:54.060528 1648 tablet_service.cc:1467] Processing CreateTablet for tablet e8b33cdd11ba431cbcde2d9ca97700df (DEFAULT_TABLE table=test-table [id=38c427f7766e4477949ea8c49318d744]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:38:54.062777 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8b33cdd11ba431cbcde2d9ca97700df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:38:54.063330 2017 tablet_bootstrap.cc:492] T e8b33cdd11ba431cbcde2d9ca97700df P d97dfed0caf14d53b8a09d9b161c1e9a: Bootstrap starting.
I20260925 04:38:54.063413 2042 ts_tablet_manager.cc:1905] T 9ba1e12fb74b46a4a351bcbb97a6f71f P 04a6633bea0742fe8224838b7e84777b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:38:54.064239 2017 tablet_bootstrap.cc:654] T e8b33cdd11ba431cbcde2d9ca97700df P d97dfed0caf14d53b8a09d9b161c1e9a: Neither blocks nor log segments found. Creating new log.
I20260925 04:38:54.064788 2018 tablet_bootstrap.cc:492] T e8b33cdd11ba431cbcde2d9ca97700df P e460d968d94246c18d3569e50e4b6c5e: Bootstrap starting.
I20260925 04:38:54.065089 1496 catalog_manager.cc:4841] TS e460d968d94246c18d3569e50e4b6c5e (127.0.98.1:43563): tablet 9ba1e12fb74b46a4a351bcbb97a6f71f (table test-table [id=e1c59c908c1a41778eb4f5d395ca9206]) successfully deleted
I20260925 04:38:54.065555 1928 tablet_service.cc:1467] Processing CreateTablet for tablet e8b33cdd11ba431cbcde2d9ca97700df (DEFAULT_TABLE table=test-table [id=38c427f7766e4477949ea8c49318d744]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:38:54.065659 2018 tablet_bootstrap.cc:654] T e8b33cdd11ba431cbcde2d9ca97700df P e460d968d94246c18d3569e50e4b6c5e: Neither blocks nor log segments found. Creating new log.
I20260925 04:38:54.065786 1928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8b33cdd11ba431cbcde2d9ca97700df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:38:54.065943 2017 tablet_bootstrap.cc:492] T e8b33cdd11ba431cbcde2d9ca97700df P d97dfed0caf14d53b8a09d9b161c1e9a: No bootstrap required, opened a new log
I20260925 04:38:54.065940 2042 ts_tablet_manager.cc:1918] T 9ba1e12fb74b46a4a351bcbb97a6f71f P 04a6633bea0742fe8224838b7e84777b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:38:54.066016 2017 ts_tablet_manager.cc:1397] T e8b33cdd11ba431cbcde2d9ca97700df P d97dfed0caf14d53b8a09d9b161c1e9a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 04:38:54.066040 2042 log.cc:1196] T 9ba1e12fb74b46a4a351bcbb97a6f71f P 04a6633bea0742fe8224838b7e84777b: Deleting WAL directory at /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/ts-2/wal/wals/9ba1e12fb74b46a4a351bcbb97a6f71f
I20260925 04:38:54.066313 2042 ts_tablet_manager.cc:1939] T 9ba1e12fb74b46a4a351bcbb97a6f71f P 04a6633bea0742fe8224838b7e84777b: Deleting consensus metadata
I20260925 04:38:54.066468 2017 raft_consensus.cc:348] T e8b33cdd11ba431cbcde2d9ca97700df P d97dfed0caf14d53b8a09d9b161c1e9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e460d968d94246c18d3569e50e4b6c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43563 } } peers { permanent_uuid: "d97dfed0caf14d53b8a09d9b161c1e9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32931 } } peers { permanent_uuid: "04a6633bea0742fe8224838b7e84777b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34353 } }
I20260925 04:38:54.066582 2017 raft_consensus.cc:374] T e8b33cdd11ba431cbcde2d9ca97700df P d97dfed0caf14d53b8a09d9b161c1e9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:38:54.066613 2017 raft_consensus.cc:729] T e8b33cdd11ba431cbcde2d9ca97700df P d97dfed0caf14d53b8a09d9b161c1e9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d97dfed0caf14d53b8a09d9b161c1e9a, State: Initialized, Role: FOLLOWER
I20260925 04:38:54.066730 2017 consensus_queue.cc:260] T e8b33cdd11ba431cbcde2d9ca97700df P d97dfed0caf14d53b8a09d9b161c1e9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e460d968d94246c18d3569e50e4b6c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43563 } } peers { permanent_uuid: "d97dfed0caf14d53b8a09d9b161c1e9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32931 } } peers { permanent_uuid: "04a6633bea0742fe8224838b7e84777b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34353 } }
I20260925 04:38:54.066962 2017 ts_tablet_manager.cc:1428] T e8b33cdd11ba431cbcde2d9ca97700df P d97dfed0caf14d53b8a09d9b161c1e9a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 04:38:54.067870 2019 tablet_bootstrap.cc:492] T e8b33cdd11ba431cbcde2d9ca97700df P 04a6633bea0742fe8224838b7e84777b: Bootstrap starting.
I20260925 04:38:54.068502 2018 tablet_bootstrap.cc:492] T e8b33cdd11ba431cbcde2d9ca97700df P e460d968d94246c18d3569e50e4b6c5e: No bootstrap required, opened a new log
I20260925 04:38:54.068588 2018 ts_tablet_manager.cc:1397] T e8b33cdd11ba431cbcde2d9ca97700df P e460d968d94246c18d3569e50e4b6c5e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 04:38:54.068796 2019 tablet_bootstrap.cc:654] T e8b33cdd11ba431cbcde2d9ca97700df P 04a6633bea0742fe8224838b7e84777b: Neither blocks nor log segments found. Creating new log.
I20260925 04:38:54.069037 2018 raft_consensus.cc:348] T e8b33cdd11ba431cbcde2d9ca97700df P e460d968d94246c18d3569e50e4b6c5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e460d968d94246c18d3569e50e4b6c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43563 } } peers { permanent_uuid: "d97dfed0caf14d53b8a09d9b161c1e9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32931 } } peers { permanent_uuid: "04a6633bea0742fe8224838b7e84777b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34353 } }
I20260925 04:38:54.069164 2018 raft_consensus.cc:374] T e8b33cdd11ba431cbcde2d9ca97700df P e460d968d94246c18d3569e50e4b6c5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:38:54.069203 2018 raft_consensus.cc:729] T e8b33cdd11ba431cbcde2d9ca97700df P e460d968d94246c18d3569e50e4b6c5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e460d968d94246c18d3569e50e4b6c5e, State: Initialized, Role: FOLLOWER
I20260925 04:38:54.069329 2018 consensus_queue.cc:260] T e8b33cdd11ba431cbcde2d9ca97700df P e460d968d94246c18d3569e50e4b6c5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e460d968d94246c18d3569e50e4b6c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43563 } } peers { permanent_uuid: "d97dfed0caf14d53b8a09d9b161c1e9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32931 } } peers { permanent_uuid: "04a6633bea0742fe8224838b7e84777b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34353 } }
I20260925 04:38:54.069585 2018 ts_tablet_manager.cc:1428] T e8b33cdd11ba431cbcde2d9ca97700df P e460d968d94246c18d3569e50e4b6c5e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 04:38:54.069732 1495 catalog_manager.cc:4841] TS 04a6633bea0742fe8224838b7e84777b (127.0.98.3:34353): tablet 9ba1e12fb74b46a4a351bcbb97a6f71f (table test-table [id=e1c59c908c1a41778eb4f5d395ca9206]) successfully deleted
I20260925 04:38:54.070642 2019 tablet_bootstrap.cc:492] T e8b33cdd11ba431cbcde2d9ca97700df P 04a6633bea0742fe8224838b7e84777b: No bootstrap required, opened a new log
I20260925 04:38:54.070724 2019 ts_tablet_manager.cc:1397] T e8b33cdd11ba431cbcde2d9ca97700df P 04a6633bea0742fe8224838b7e84777b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 04:38:54.071133 2019 raft_consensus.cc:348] T e8b33cdd11ba431cbcde2d9ca97700df P 04a6633bea0742fe8224838b7e84777b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e460d968d94246c18d3569e50e4b6c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43563 } } peers { permanent_uuid: "d97dfed0caf14d53b8a09d9b161c1e9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32931 } } peers { permanent_uuid: "04a6633bea0742fe8224838b7e84777b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34353 } }
I20260925 04:38:54.071225 2019 raft_consensus.cc:374] T e8b33cdd11ba431cbcde2d9ca97700df P 04a6633bea0742fe8224838b7e84777b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:38:54.071254 2019 raft_consensus.cc:729] T e8b33cdd11ba431cbcde2d9ca97700df P 04a6633bea0742fe8224838b7e84777b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04a6633bea0742fe8224838b7e84777b, State: Initialized, Role: FOLLOWER
I20260925 04:38:54.071368 2019 consensus_queue.cc:260] T e8b33cdd11ba431cbcde2d9ca97700df P 04a6633bea0742fe8224838b7e84777b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e460d968d94246c18d3569e50e4b6c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43563 } } peers { permanent_uuid: "d97dfed0caf14d53b8a09d9b161c1e9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32931 } } peers { permanent_uuid: "04a6633bea0742fe8224838b7e84777b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34353 } }
I20260925 04:38:54.071661 2019 ts_tablet_manager.cc:1428] T e8b33cdd11ba431cbcde2d9ca97700df P 04a6633bea0742fe8224838b7e84777b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260925 04:38:54.083652 1995 tablet.cc:2369] T e8b33cdd11ba431cbcde2d9ca97700df P 04a6633bea0742fe8224838b7e84777b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 04:38:54.105569 1855 tablet.cc:2369] T e8b33cdd11ba431cbcde2d9ca97700df P d97dfed0caf14d53b8a09d9b161c1e9a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 04:38:54.107183 1715 tablet.cc:2369] T e8b33cdd11ba431cbcde2d9ca97700df P e460d968d94246c18d3569e50e4b6c5e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:38:54.291405 2023 raft_consensus.cc:482] T e8b33cdd11ba431cbcde2d9ca97700df P e460d968d94246c18d3569e50e4b6c5e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:38:54.291585 2023 raft_consensus.cc:504] T e8b33cdd11ba431cbcde2d9ca97700df P e460d968d94246c18d3569e50e4b6c5e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e460d968d94246c18d3569e50e4b6c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43563 } } peers { permanent_uuid: "d97dfed0caf14d53b8a09d9b161c1e9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32931 } } peers { permanent_uuid: "04a6633bea0742fe8224838b7e84777b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34353 } }
I20260925 04:38:54.292708 2023 leader_election.cc:288] T e8b33cdd11ba431cbcde2d9ca97700df P e460d968d94246c18d3569e50e4b6c5e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d97dfed0caf14d53b8a09d9b161c1e9a (127.0.98.2:32931), 04a6633bea0742fe8224838b7e84777b (127.0.98.3:34353)
I20260925 04:38:54.296862 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:54.293097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53137 (local address 127.0.98.2:32931)
0925 04:38:54.293248 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:54.293252 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:54.293477 (+ 225us) server_negotiation.cc:407] Connection header received
0925 04:38:54.293598 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:54.293601 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:54.293646 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:54.293709 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:38:54.293996 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:54.294511 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:54.295816 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:54.296218 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:54.296312 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:54.296379 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:54.296642 (+ 263us) server_negotiation.cc:299] Negotiation successful
0925 04:38:54.296724 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 04:38:54.296862 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:54.293189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45383 (local address 127.0.98.3:34353)
0925 04:38:54.293337 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:54.293341 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:54.293541 (+ 200us) server_negotiation.cc:407] Connection header received
0925 04:38:54.293673 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:54.293676 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:54.293716 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:54.293777 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:38:54.294041 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:54.294562 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:54.295799 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:54.296213 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:54.296312 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:54.296381 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:54.296647 (+ 266us) server_negotiation.cc:299] Negotiation successful
0925 04:38:54.296728 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 04:38:54.296885 2046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:54.293057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34353 (local address 127.0.98.1:45383)
0925 04:38:54.293447 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:54.293458 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 04:38:54.293570 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:38:54.293804 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:38:54.293807 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:38:54.293822 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:38:54.293939 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:54.293944 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:54.294675 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:54.294678 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:54.295681 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:54.295692 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:54.296352 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:54.296356 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:54.296531 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:54.296552 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 04:38:54.296619 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 04:38:54.296735 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"mutex_wait_us":107,"thread_start_us":93,"threads_started":1}
I20260925 04:38:54.296891 2047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:54.293011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32931 (local address 127.0.98.1:53137)
0925 04:38:54.293369 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:54.293381 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:38:54.293506 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:38:54.293737 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:38:54.293739 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:38:54.293755 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:38:54.293888 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:54.293893 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:54.294626 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:54.294630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:54.295679 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:54.295692 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:54.296357 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:54.296361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:54.296502 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:54.296539 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 04:38:54.296608 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 04:38:54.296718 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":138,"thread_start_us":93,"threads_started":1}
I20260925 04:38:54.297544 1933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b33cdd11ba431cbcde2d9ca97700df" candidate_uuid: "e460d968d94246c18d3569e50e4b6c5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04a6633bea0742fe8224838b7e84777b" is_pre_election: true
I20260925 04:38:54.297595 1808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b33cdd11ba431cbcde2d9ca97700df" candidate_uuid: "e460d968d94246c18d3569e50e4b6c5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d97dfed0caf14d53b8a09d9b161c1e9a" is_pre_election: true
I20260925 04:38:54.297734 1808 raft_consensus.cc:2454] T e8b33cdd11ba431cbcde2d9ca97700df P d97dfed0caf14d53b8a09d9b161c1e9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e460d968d94246c18d3569e50e4b6c5e in term 0.
I20260925 04:38:54.297780 1933 raft_consensus.cc:2454] T e8b33cdd11ba431cbcde2d9ca97700df P 04a6633bea0742fe8224838b7e84777b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e460d968d94246c18d3569e50e4b6c5e in term 0.
I20260925 04:38:54.298122 1602 leader_election.cc:302] T e8b33cdd11ba431cbcde2d9ca97700df P e460d968d94246c18d3569e50e4b6c5e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 04a6633bea0742fe8224838b7e84777b, d97dfed0caf14d53b8a09d9b161c1e9a, e460d968d94246c18d3569e50e4b6c5e; no voters:
I20260925 04:38:54.298400 2023 raft_consensus.cc:2781] T e8b33cdd11ba431cbcde2d9ca97700df P e460d968d94246c18d3569e50e4b6c5e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:38:54.298465 2023 raft_consensus.cc:482] T e8b33cdd11ba431cbcde2d9ca97700df P e460d968d94246c18d3569e50e4b6c5e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:38:54.298506 2023 raft_consensus.cc:3037] T e8b33cdd11ba431cbcde2d9ca97700df P e460d968d94246c18d3569e50e4b6c5e [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:38:54.299515 2023 raft_consensus.cc:504] T e8b33cdd11ba431cbcde2d9ca97700df P e460d968d94246c18d3569e50e4b6c5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e460d968d94246c18d3569e50e4b6c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43563 } } peers { permanent_uuid: "d97dfed0caf14d53b8a09d9b161c1e9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32931 } } peers { permanent_uuid: "04a6633bea0742fe8224838b7e84777b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34353 } }
I20260925 04:38:54.299911 2023 leader_election.cc:288] T e8b33cdd11ba431cbcde2d9ca97700df P e460d968d94246c18d3569e50e4b6c5e [CANDIDATE]: Term 1 election: Requested vote from peers d97dfed0caf14d53b8a09d9b161c1e9a (127.0.98.2:32931), 04a6633bea0742fe8224838b7e84777b (127.0.98.3:34353)
I20260925 04:38:54.300271 1808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b33cdd11ba431cbcde2d9ca97700df" candidate_uuid: "e460d968d94246c18d3569e50e4b6c5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d97dfed0caf14d53b8a09d9b161c1e9a"
I20260925 04:38:54.300278 1933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b33cdd11ba431cbcde2d9ca97700df" candidate_uuid: "e460d968d94246c18d3569e50e4b6c5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04a6633bea0742fe8224838b7e84777b"
I20260925 04:38:54.300379 1808 raft_consensus.cc:3037] T e8b33cdd11ba431cbcde2d9ca97700df P d97dfed0caf14d53b8a09d9b161c1e9a [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:38:54.300405 1933 raft_consensus.cc:3037] T e8b33cdd11ba431cbcde2d9ca97700df P 04a6633bea0742fe8224838b7e84777b [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:38:54.301366 1808 raft_consensus.cc:2454] T e8b33cdd11ba431cbcde2d9ca97700df P d97dfed0caf14d53b8a09d9b161c1e9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e460d968d94246c18d3569e50e4b6c5e in term 1.
I20260925 04:38:54.301513 1933 raft_consensus.cc:2454] T e8b33cdd11ba431cbcde2d9ca97700df P 04a6633bea0742fe8224838b7e84777b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e460d968d94246c18d3569e50e4b6c5e in term 1.
I20260925 04:38:54.301692 1602 leader_election.cc:302] T e8b33cdd11ba431cbcde2d9ca97700df P e460d968d94246c18d3569e50e4b6c5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d97dfed0caf14d53b8a09d9b161c1e9a, e460d968d94246c18d3569e50e4b6c5e; no voters:
I20260925 04:38:54.301862 2023 raft_consensus.cc:2781] T e8b33cdd11ba431cbcde2d9ca97700df P e460d968d94246c18d3569e50e4b6c5e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:38:54.302088 2023 raft_consensus.cc:686] T e8b33cdd11ba431cbcde2d9ca97700df P e460d968d94246c18d3569e50e4b6c5e [term 1 LEADER]: Becoming Leader. State: Replica: e460d968d94246c18d3569e50e4b6c5e, State: Running, Role: LEADER
I20260925 04:38:54.302336 2023 consensus_queue.cc:237] T e8b33cdd11ba431cbcde2d9ca97700df P e460d968d94246c18d3569e50e4b6c5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e460d968d94246c18d3569e50e4b6c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43563 } } peers { permanent_uuid: "d97dfed0caf14d53b8a09d9b161c1e9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32931 } } peers { permanent_uuid: "04a6633bea0742fe8224838b7e84777b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34353 } }
I20260925 04:38:54.304666 1510 catalog_manager.cc:5495] T e8b33cdd11ba431cbcde2d9ca97700df P e460d968d94246c18d3569e50e4b6c5e reported cstate change: term changed from 0 to 1, leader changed from <none> to e460d968d94246c18d3569e50e4b6c5e (127.0.98.1). New cstate: current_term: 1 leader_uuid: "e460d968d94246c18d3569e50e4b6c5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e460d968d94246c18d3569e50e4b6c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d97dfed0caf14d53b8a09d9b161c1e9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04a6633bea0742fe8224838b7e84777b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34353 } health_report { overall_health: UNKNOWN } } }
Sep 25 04:38:54 dist-test-slave-dc95 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311133, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:38:54.346124 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:54.337422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37348 (local address 127.0.98.62:46757)
0925 04:38:54.337594 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:54.337600 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:54.337827 (+ 227us) server_negotiation.cc:407] Connection header received
0925 04:38:54.338006 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:54.338009 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:54.338057 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:54.338126 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:54.338854 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:54.339359 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:54.340215 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:54.340418 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:54.340502 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:54.342860 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:38:54.342863 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:38:54.342865 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:38:54.342887 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:38:54.344279 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:54.344781 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:54.344785 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:54.344786 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:54.344868 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:54.345190 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:54.345193 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:54.345194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:54.345402 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:38:54.345486 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:54.345820 (+ 334us) server_negotiation.cc:299] Negotiation successful
0925 04:38:54.345946 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":66}
I20260925 04:38:54.346127 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:54.337422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46757 (local address 127.0.0.1:37348)
0925 04:38:54.337740 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:54.337753 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:38:54.337869 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:38:54.338159 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:38:54.338161 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:38:54.338642 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:38:54.338738 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:54.338745 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:54.339485 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:54.339489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:54.340088 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:54.340126 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:54.340515 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:54.340518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:54.340637 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:54.341203 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:38:54.341221 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:38:54.342725 (+ 1504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:38:54.344406 (+ 1681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:54.344416 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:54.344421 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:54.344671 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:54.345002 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:54.345005 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:54.345007 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:54.345089 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:54.345519 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:38:54.345530 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:38:54.345649 (+ 119us) client_negotiation.cc:769] Sending connection context
0925 04:38:54.345751 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 04:38:54.345896 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":31,"thread_start_us":129,"threads_started":1}
I20260925 04:38:54.350281 1510 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37348:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 04:38:54.351069 1510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:38:54.356297 1648 tablet_service.cc:1467] Processing CreateTablet for tablet c0be15232ba443be95cc53addb34617d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1f8a62c6aee94263922f23a0ee8bc6bc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 04:38:54.356371 1788 tablet_service.cc:1467] Processing CreateTablet for tablet c0be15232ba443be95cc53addb34617d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1f8a62c6aee94263922f23a0ee8bc6bc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 04:38:54.356732 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0be15232ba443be95cc53addb34617d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:38:54.356776 1788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0be15232ba443be95cc53addb34617d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:38:54.358318 1928 tablet_service.cc:1467] Processing CreateTablet for tablet c0be15232ba443be95cc53addb34617d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1f8a62c6aee94263922f23a0ee8bc6bc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 04:38:54.358803 1928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0be15232ba443be95cc53addb34617d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:38:54.359467 2018 tablet_bootstrap.cc:492] T c0be15232ba443be95cc53addb34617d P e460d968d94246c18d3569e50e4b6c5e: Bootstrap starting.
I20260925 04:38:54.360406 2017 tablet_bootstrap.cc:492] T c0be15232ba443be95cc53addb34617d P d97dfed0caf14d53b8a09d9b161c1e9a: Bootstrap starting.
I20260925 04:38:54.360440 2018 tablet_bootstrap.cc:654] T c0be15232ba443be95cc53addb34617d P e460d968d94246c18d3569e50e4b6c5e: Neither blocks nor log segments found. Creating new log.
I20260925 04:38:54.361294 2017 tablet_bootstrap.cc:654] T c0be15232ba443be95cc53addb34617d P d97dfed0caf14d53b8a09d9b161c1e9a: Neither blocks nor log segments found. Creating new log.
I20260925 04:38:54.361747 2019 tablet_bootstrap.cc:492] T c0be15232ba443be95cc53addb34617d P 04a6633bea0742fe8224838b7e84777b: Bootstrap starting.
I20260925 04:38:54.361924 2018 tablet_bootstrap.cc:492] T c0be15232ba443be95cc53addb34617d P e460d968d94246c18d3569e50e4b6c5e: No bootstrap required, opened a new log
I20260925 04:38:54.362002 2018 ts_tablet_manager.cc:1397] T c0be15232ba443be95cc53addb34617d P e460d968d94246c18d3569e50e4b6c5e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 04:38:54.362385 2018 raft_consensus.cc:348] T c0be15232ba443be95cc53addb34617d P e460d968d94246c18d3569e50e4b6c5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e460d968d94246c18d3569e50e4b6c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43563 } } peers { permanent_uuid: "d97dfed0caf14d53b8a09d9b161c1e9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32931 } } peers { permanent_uuid: "04a6633bea0742fe8224838b7e84777b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34353 } }
I20260925 04:38:54.362474 2018 raft_consensus.cc:374] T c0be15232ba443be95cc53addb34617d P e460d968d94246c18d3569e50e4b6c5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:38:54.362494 2018 raft_consensus.cc:729] T c0be15232ba443be95cc53addb34617d P e460d968d94246c18d3569e50e4b6c5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e460d968d94246c18d3569e50e4b6c5e, State: Initialized, Role: FOLLOWER
I20260925 04:38:54.362609 2018 consensus_queue.cc:260] T c0be15232ba443be95cc53addb34617d P e460d968d94246c18d3569e50e4b6c5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e460d968d94246c18d3569e50e4b6c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43563 } } peers { permanent_uuid: "d97dfed0caf14d53b8a09d9b161c1e9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32931 } } peers { permanent_uuid: "04a6633bea0742fe8224838b7e84777b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34353 } }
I20260925 04:38:54.362637 2019 tablet_bootstrap.cc:654] T c0be15232ba443be95cc53addb34617d P 04a6633bea0742fe8224838b7e84777b: Neither blocks nor log segments found. Creating new log.
I20260925 04:38:54.362865 2018 ts_tablet_manager.cc:1428] T c0be15232ba443be95cc53addb34617d P e460d968d94246c18d3569e50e4b6c5e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 04:38:54.362831 2023 tablet_replica.cc:439] T c0be15232ba443be95cc53addb34617d P e460d968d94246c18d3569e50e4b6c5e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e460d968d94246c18d3569e50e4b6c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43563 } } peers { permanent_uuid: "d97dfed0caf14d53b8a09d9b161c1e9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32931 } } peers { permanent_uuid: "04a6633bea0742fe8224838b7e84777b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34353 } } }
I20260925 04:38:54.362948 2023 tablet_replica.cc:442] T c0be15232ba443be95cc53addb34617d P e460d968d94246c18d3569e50e4b6c5e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:38:54.364395 2019 tablet_bootstrap.cc:492] T c0be15232ba443be95cc53addb34617d P 04a6633bea0742fe8224838b7e84777b: No bootstrap required, opened a new log
I20260925 04:38:54.364481 2019 ts_tablet_manager.cc:1397] T c0be15232ba443be95cc53addb34617d P 04a6633bea0742fe8224838b7e84777b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 04:38:54.364904 2019 raft_consensus.cc:348] T c0be15232ba443be95cc53addb34617d P 04a6633bea0742fe8224838b7e84777b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e460d968d94246c18d3569e50e4b6c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43563 } } peers { permanent_uuid: "d97dfed0caf14d53b8a09d9b161c1e9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32931 } } peers { permanent_uuid: "04a6633bea0742fe8224838b7e84777b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34353 } }
I20260925 04:38:54.365020 2019 raft_consensus.cc:374] T c0be15232ba443be95cc53addb34617d P 04a6633bea0742fe8224838b7e84777b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:38:54.365046 2019 raft_consensus.cc:729] T c0be15232ba443be95cc53addb34617d P 04a6633bea0742fe8224838b7e84777b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04a6633bea0742fe8224838b7e84777b, State: Initialized, Role: FOLLOWER
I20260925 04:38:54.365154 2019 consensus_queue.cc:260] T c0be15232ba443be95cc53addb34617d P 04a6633bea0742fe8224838b7e84777b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e460d968d94246c18d3569e50e4b6c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43563 } } peers { permanent_uuid: "d97dfed0caf14d53b8a09d9b161c1e9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32931 } } peers { permanent_uuid: "04a6633bea0742fe8224838b7e84777b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34353 } }
I20260925 04:38:54.365398 2019 ts_tablet_manager.cc:1428] T c0be15232ba443be95cc53addb34617d P 04a6633bea0742fe8224838b7e84777b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 04:38:54.365376 2035 tablet_replica.cc:439] T c0be15232ba443be95cc53addb34617d P 04a6633bea0742fe8224838b7e84777b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e460d968d94246c18d3569e50e4b6c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43563 } } peers { permanent_uuid: "d97dfed0caf14d53b8a09d9b161c1e9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32931 } } peers { permanent_uuid: "04a6633bea0742fe8224838b7e84777b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34353 } } }
I20260925 04:38:54.365561 2035 tablet_replica.cc:442] T c0be15232ba443be95cc53addb34617d P 04a6633bea0742fe8224838b7e84777b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:38:54.370589 2017 tablet_bootstrap.cc:492] T c0be15232ba443be95cc53addb34617d P d97dfed0caf14d53b8a09d9b161c1e9a: No bootstrap required, opened a new log
I20260925 04:38:54.370694 2017 ts_tablet_manager.cc:1397] T c0be15232ba443be95cc53addb34617d P d97dfed0caf14d53b8a09d9b161c1e9a: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20260925 04:38:54.371181 2017 raft_consensus.cc:348] T c0be15232ba443be95cc53addb34617d P d97dfed0caf14d53b8a09d9b161c1e9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e460d968d94246c18d3569e50e4b6c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43563 } } peers { permanent_uuid: "d97dfed0caf14d53b8a09d9b161c1e9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32931 } } peers { permanent_uuid: "04a6633bea0742fe8224838b7e84777b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34353 } }
I20260925 04:38:54.371301 2017 raft_consensus.cc:374] T c0be15232ba443be95cc53addb34617d P d97dfed0caf14d53b8a09d9b161c1e9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:38:54.371323 2017 raft_consensus.cc:729] T c0be15232ba443be95cc53addb34617d P d97dfed0caf14d53b8a09d9b161c1e9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d97dfed0caf14d53b8a09d9b161c1e9a, State: Initialized, Role: FOLLOWER
I20260925 04:38:54.371474 2017 consensus_queue.cc:260] T c0be15232ba443be95cc53addb34617d P d97dfed0caf14d53b8a09d9b161c1e9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e460d968d94246c18d3569e50e4b6c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43563 } } peers { permanent_uuid: "d97dfed0caf14d53b8a09d9b161c1e9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32931 } } peers { permanent_uuid: "04a6633bea0742fe8224838b7e84777b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34353 } }
I20260925 04:38:54.371783 2017 ts_tablet_manager.cc:1428] T c0be15232ba443be95cc53addb34617d P d97dfed0caf14d53b8a09d9b161c1e9a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 04:38:54.371739 2024 tablet_replica.cc:439] T c0be15232ba443be95cc53addb34617d P d97dfed0caf14d53b8a09d9b161c1e9a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e460d968d94246c18d3569e50e4b6c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43563 } } peers { permanent_uuid: "d97dfed0caf14d53b8a09d9b161c1e9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32931 } } peers { permanent_uuid: "04a6633bea0742fe8224838b7e84777b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34353 } } }
I20260925 04:38:54.371886 2024 tablet_replica.cc:442] T c0be15232ba443be95cc53addb34617d P d97dfed0caf14d53b8a09d9b161c1e9a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:38:54.558102 2024 raft_consensus.cc:482] T c0be15232ba443be95cc53addb34617d P d97dfed0caf14d53b8a09d9b161c1e9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:38:54.558274 2024 raft_consensus.cc:504] T c0be15232ba443be95cc53addb34617d P d97dfed0caf14d53b8a09d9b161c1e9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e460d968d94246c18d3569e50e4b6c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43563 } } peers { permanent_uuid: "d97dfed0caf14d53b8a09d9b161c1e9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32931 } } peers { permanent_uuid: "04a6633bea0742fe8224838b7e84777b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34353 } }
I20260925 04:38:54.559440 2024 leader_election.cc:288] T c0be15232ba443be95cc53addb34617d P d97dfed0caf14d53b8a09d9b161c1e9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e460d968d94246c18d3569e50e4b6c5e (127.0.98.1:43563), 04a6633bea0742fe8224838b7e84777b (127.0.98.3:34353)
I20260925 04:38:54.564129 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:54.559844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46923 (local address 127.0.98.1:43563)
0925 04:38:54.560066 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:54.560071 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:54.560188 (+ 117us) server_negotiation.cc:407] Connection header received
0925 04:38:54.560307 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:54.560310 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:54.560356 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:54.560427 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:38:54.560730 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:54.561255 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:54.563032 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:54.563536 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:54.563682 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:54.563753 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:54.563911 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 04:38:54.563990 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":54,"threads_started":1}
I20260925 04:38:54.564162 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:54.559806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43263 (local address 127.0.98.3:34353)
0925 04:38:54.559936 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:54.559940 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:54.560168 (+ 228us) server_negotiation.cc:407] Connection header received
0925 04:38:54.560316 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:54.560319 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:54.560364 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:54.560430 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:38:54.560756 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:54.561241 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:54.563067 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:54.563576 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:54.563718 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:54.563782 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:54.563954 (+ 172us) server_negotiation.cc:299] Negotiation successful
0925 04:38:54.564048 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 04:38:54.564167 2060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:54.559774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43563 (local address 127.0.98.2:46923)
0925 04:38:54.560057 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:54.560069 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:38:54.560205 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:38:54.560449 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:38:54.560452 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:38:54.560473 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:38:54.560622 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:54.560628 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:54.561368 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:54.561371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:54.562881 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:54.562895 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:54.563684 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:54.563688 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:54.563794 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:54.563817 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 04:38:54.563883 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 04:38:54.563966 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":106,"spinlock_wait_cycles":32640,"thread_start_us":69,"threads_started":1}
I20260925 04:38:54.564181 2059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:54.559814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34353 (local address 127.0.98.2:43263)
0925 04:38:54.560063 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:54.560077 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:38:54.560213 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:38:54.560453 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:38:54.560456 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:38:54.560476 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:38:54.560646 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:54.560652 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:54.561378 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:54.561381 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:54.562881 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:54.562895 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:54.563720 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:54.563723 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:54.563839 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:54.563856 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 04:38:54.563921 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 04:38:54.563996 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":119,"spinlock_wait_cycles":2816,"thread_start_us":72,"threads_started":1}
I20260925 04:38:54.564671 1933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0be15232ba443be95cc53addb34617d" candidate_uuid: "d97dfed0caf14d53b8a09d9b161c1e9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04a6633bea0742fe8224838b7e84777b" is_pre_election: true
I20260925 04:38:54.564813 1933 raft_consensus.cc:2454] T c0be15232ba443be95cc53addb34617d P 04a6633bea0742fe8224838b7e84777b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d97dfed0caf14d53b8a09d9b161c1e9a in term 0.
I20260925 04:38:54.564937 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0be15232ba443be95cc53addb34617d" candidate_uuid: "d97dfed0caf14d53b8a09d9b161c1e9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e460d968d94246c18d3569e50e4b6c5e" is_pre_election: true
I20260925 04:38:54.565073 1668 raft_consensus.cc:2454] T c0be15232ba443be95cc53addb34617d P e460d968d94246c18d3569e50e4b6c5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d97dfed0caf14d53b8a09d9b161c1e9a in term 0.
I20260925 04:38:54.565201 1739 leader_election.cc:302] T c0be15232ba443be95cc53addb34617d P d97dfed0caf14d53b8a09d9b161c1e9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04a6633bea0742fe8224838b7e84777b, d97dfed0caf14d53b8a09d9b161c1e9a; no voters:
I20260925 04:38:54.565472 2024 raft_consensus.cc:2781] T c0be15232ba443be95cc53addb34617d P d97dfed0caf14d53b8a09d9b161c1e9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:38:54.565527 2024 raft_consensus.cc:482] T c0be15232ba443be95cc53addb34617d P d97dfed0caf14d53b8a09d9b161c1e9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:38:54.565572 2024 raft_consensus.cc:3037] T c0be15232ba443be95cc53addb34617d P d97dfed0caf14d53b8a09d9b161c1e9a [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:38:54.566532 2024 raft_consensus.cc:504] T c0be15232ba443be95cc53addb34617d P d97dfed0caf14d53b8a09d9b161c1e9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e460d968d94246c18d3569e50e4b6c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43563 } } peers { permanent_uuid: "d97dfed0caf14d53b8a09d9b161c1e9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32931 } } peers { permanent_uuid: "04a6633bea0742fe8224838b7e84777b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34353 } }
I20260925 04:38:54.566866 2024 leader_election.cc:288] T c0be15232ba443be95cc53addb34617d P d97dfed0caf14d53b8a09d9b161c1e9a [CANDIDATE]: Term 1 election: Requested vote from peers e460d968d94246c18d3569e50e4b6c5e (127.0.98.1:43563), 04a6633bea0742fe8224838b7e84777b (127.0.98.3:34353)
I20260925 04:38:54.567248 1933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0be15232ba443be95cc53addb34617d" candidate_uuid: "d97dfed0caf14d53b8a09d9b161c1e9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04a6633bea0742fe8224838b7e84777b"
I20260925 04:38:54.567265 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0be15232ba443be95cc53addb34617d" candidate_uuid: "d97dfed0caf14d53b8a09d9b161c1e9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e460d968d94246c18d3569e50e4b6c5e"
I20260925 04:38:54.567363 1933 raft_consensus.cc:3037] T c0be15232ba443be95cc53addb34617d P 04a6633bea0742fe8224838b7e84777b [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:38:54.567407 1668 raft_consensus.cc:3037] T c0be15232ba443be95cc53addb34617d P e460d968d94246c18d3569e50e4b6c5e [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:38:54.568336 1933 raft_consensus.cc:2454] T c0be15232ba443be95cc53addb34617d P 04a6633bea0742fe8224838b7e84777b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d97dfed0caf14d53b8a09d9b161c1e9a in term 1.
I20260925 04:38:54.568426 1668 raft_consensus.cc:2454] T c0be15232ba443be95cc53addb34617d P e460d968d94246c18d3569e50e4b6c5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d97dfed0caf14d53b8a09d9b161c1e9a in term 1.
I20260925 04:38:54.568677 1739 leader_election.cc:302] T c0be15232ba443be95cc53addb34617d P d97dfed0caf14d53b8a09d9b161c1e9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04a6633bea0742fe8224838b7e84777b, d97dfed0caf14d53b8a09d9b161c1e9a; no voters:
I20260925 04:38:54.568876 2024 raft_consensus.cc:2781] T c0be15232ba443be95cc53addb34617d P d97dfed0caf14d53b8a09d9b161c1e9a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:38:54.569048 2024 raft_consensus.cc:686] T c0be15232ba443be95cc53addb34617d P d97dfed0caf14d53b8a09d9b161c1e9a [term 1 LEADER]: Becoming Leader. State: Replica: d97dfed0caf14d53b8a09d9b161c1e9a, State: Running, Role: LEADER
I20260925 04:38:54.569326 2024 consensus_queue.cc:237] T c0be15232ba443be95cc53addb34617d P d97dfed0caf14d53b8a09d9b161c1e9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e460d968d94246c18d3569e50e4b6c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43563 } } peers { permanent_uuid: "d97dfed0caf14d53b8a09d9b161c1e9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32931 } } peers { permanent_uuid: "04a6633bea0742fe8224838b7e84777b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34353 } }
I20260925 04:38:54.570583 2062 tablet_replica.cc:439] T c0be15232ba443be95cc53addb34617d P d97dfed0caf14d53b8a09d9b161c1e9a: TxnStatusTablet state changed. Reason: New leader d97dfed0caf14d53b8a09d9b161c1e9a. Latest consensus state: current_term: 1 leader_uuid: "d97dfed0caf14d53b8a09d9b161c1e9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e460d968d94246c18d3569e50e4b6c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43563 } } peers { permanent_uuid: "d97dfed0caf14d53b8a09d9b161c1e9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32931 } } peers { permanent_uuid: "04a6633bea0742fe8224838b7e84777b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34353 } } }
I20260925 04:38:54.570716 2062 tablet_replica.cc:442] T c0be15232ba443be95cc53addb34617d P d97dfed0caf14d53b8a09d9b161c1e9a: This TxnStatusTablet replica's current role is: LEADER
I20260925 04:38:54.570930 2064 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 04:38:54.571071 2064 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 04:38:54.571936 1510 catalog_manager.cc:5495] T c0be15232ba443be95cc53addb34617d P d97dfed0caf14d53b8a09d9b161c1e9a reported cstate change: term changed from 0 to 1, leader changed from <none> to d97dfed0caf14d53b8a09d9b161c1e9a (127.0.98.2). New cstate: current_term: 1 leader_uuid: "d97dfed0caf14d53b8a09d9b161c1e9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e460d968d94246c18d3569e50e4b6c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d97dfed0caf14d53b8a09d9b161c1e9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "04a6633bea0742fe8224838b7e84777b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34353 } health_report { overall_health: UNKNOWN } } }
I20260925 04:38:54.607167 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:54.603129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39322 (local address 127.0.98.2:32931)
0925 04:38:54.603241 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:54.603245 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:54.603327 (+ 82us) server_negotiation.cc:407] Connection header received
0925 04:38:54.603789 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:54.603793 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:54.603864 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:54.603961 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:38:54.604297 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:54.604841 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:54.605719 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:54.605933 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:54.606023 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:54.606324 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:38:54.606747 (+ 423us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:38:54.606849 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:54.606955 (+ 106us) server_negotiation.cc:299] Negotiation successful
0925 04:38:54.607012 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 04:38:54.607167 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:54.602992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32931 (local address 127.0.0.1:39322)
0925 04:38:54.603262 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:54.603276 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:38:54.603479 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:38:54.604079 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:38:54.604083 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:38:54.604092 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 04:38:54.604182 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:54.604187 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:54.604951 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:54.604954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:54.605586 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:54.605595 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:54.606045 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:54.606048 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:54.606135 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:54.606185 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 04:38:54.606846 (+ 661us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 04:38:54.606864 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 04:38:54.606929 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 04:38:54.607031 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159}
I20260925 04:38:54.610688 1668 raft_consensus.cc:1261] T c0be15232ba443be95cc53addb34617d P e460d968d94246c18d3569e50e4b6c5e [term 1 FOLLOWER]: Refusing update from remote peer d97dfed0caf14d53b8a09d9b161c1e9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:38:54.610739 1933 raft_consensus.cc:1261] T c0be15232ba443be95cc53addb34617d P 04a6633bea0742fe8224838b7e84777b [term 1 FOLLOWER]: Refusing update from remote peer d97dfed0caf14d53b8a09d9b161c1e9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:38:54.611232 2062 consensus_queue.cc:1035] T c0be15232ba443be95cc53addb34617d P d97dfed0caf14d53b8a09d9b161c1e9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "04a6633bea0742fe8224838b7e84777b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:38:54.611400 2024 consensus_queue.cc:1035] T c0be15232ba443be95cc53addb34617d P d97dfed0caf14d53b8a09d9b161c1e9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e460d968d94246c18d3569e50e4b6c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:38:54.612973 2023 tablet_replica.cc:439] T c0be15232ba443be95cc53addb34617d P e460d968d94246c18d3569e50e4b6c5e: TxnStatusTablet state changed. Reason: New leader d97dfed0caf14d53b8a09d9b161c1e9a. Latest consensus state: current_term: 1 leader_uuid: "d97dfed0caf14d53b8a09d9b161c1e9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e460d968d94246c18d3569e50e4b6c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43563 } } peers { permanent_uuid: "d97dfed0caf14d53b8a09d9b161c1e9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32931 } } peers { permanent_uuid: "04a6633bea0742fe8224838b7e84777b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34353 } } }
I20260925 04:38:54.613106 2023 tablet_replica.cc:442] T c0be15232ba443be95cc53addb34617d P e460d968d94246c18d3569e50e4b6c5e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:38:54.613097 2035 tablet_replica.cc:439] T c0be15232ba443be95cc53addb34617d P 04a6633bea0742fe8224838b7e84777b: TxnStatusTablet state changed. Reason: New leader d97dfed0caf14d53b8a09d9b161c1e9a. Latest consensus state: current_term: 1 leader_uuid: "d97dfed0caf14d53b8a09d9b161c1e9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e460d968d94246c18d3569e50e4b6c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43563 } } peers { permanent_uuid: "d97dfed0caf14d53b8a09d9b161c1e9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32931 } } peers { permanent_uuid: "04a6633bea0742fe8224838b7e84777b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34353 } } }
I20260925 04:38:54.613198 2035 tablet_replica.cc:442] T c0be15232ba443be95cc53addb34617d P 04a6633bea0742fe8224838b7e84777b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:38:54.616091 2024 tablet_replica.cc:439] T c0be15232ba443be95cc53addb34617d P d97dfed0caf14d53b8a09d9b161c1e9a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d97dfed0caf14d53b8a09d9b161c1e9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e460d968d94246c18d3569e50e4b6c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43563 } } peers { permanent_uuid: "d97dfed0caf14d53b8a09d9b161c1e9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32931 } } peers { permanent_uuid: "04a6633bea0742fe8224838b7e84777b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34353 } } }
I20260925 04:38:54.616218 2024 tablet_replica.cc:442] T c0be15232ba443be95cc53addb34617d P d97dfed0caf14d53b8a09d9b161c1e9a: This TxnStatusTablet replica's current role is: LEADER
I20260925 04:38:54.616600 2024 tablet_replica.cc:439] T c0be15232ba443be95cc53addb34617d P d97dfed0caf14d53b8a09d9b161c1e9a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d97dfed0caf14d53b8a09d9b161c1e9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e460d968d94246c18d3569e50e4b6c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43563 } } peers { permanent_uuid: "d97dfed0caf14d53b8a09d9b161c1e9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32931 } } peers { permanent_uuid: "04a6633bea0742fe8224838b7e84777b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34353 } } }
I20260925 04:38:54.616698 2024 tablet_replica.cc:442] T c0be15232ba443be95cc53addb34617d P d97dfed0caf14d53b8a09d9b161c1e9a: This TxnStatusTablet replica's current role is: LEADER
I20260925 04:38:54.616731 2035 tablet_replica.cc:439] T c0be15232ba443be95cc53addb34617d P 04a6633bea0742fe8224838b7e84777b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d97dfed0caf14d53b8a09d9b161c1e9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e460d968d94246c18d3569e50e4b6c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43563 } } peers { permanent_uuid: "d97dfed0caf14d53b8a09d9b161c1e9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32931 } } peers { permanent_uuid: "04a6633bea0742fe8224838b7e84777b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34353 } } }
I20260925 04:38:54.616827 2035 tablet_replica.cc:442] T c0be15232ba443be95cc53addb34617d P 04a6633bea0742fe8224838b7e84777b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:38:54.616806 2069 mvcc.cc:205] Tried to move back new op lower bound from 7333114407359983616 to 7333114407198609408. Current Snapshot: MvccSnapshot[applied={T|T < 7333114407359983616}]
I20260925 04:38:54.617189 2023 tablet_replica.cc:439] T c0be15232ba443be95cc53addb34617d P e460d968d94246c18d3569e50e4b6c5e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d97dfed0caf14d53b8a09d9b161c1e9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e460d968d94246c18d3569e50e4b6c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43563 } } peers { permanent_uuid: "d97dfed0caf14d53b8a09d9b161c1e9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32931 } } peers { permanent_uuid: "04a6633bea0742fe8224838b7e84777b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34353 } } }
I20260925 04:38:54.617301 2023 tablet_replica.cc:442] T c0be15232ba443be95cc53addb34617d P e460d968d94246c18d3569e50e4b6c5e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:38:54.624907 2068 mvcc.cc:205] Tried to move back new op lower bound from 7333114407359983616 to 7333114407198609408. Current Snapshot: MvccSnapshot[applied={T|T < 7333114407359983616}]
I20260925 04:38:54.632524 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:54.628845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41618 (local address 127.0.98.1:43563)
0925 04:38:54.628990 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:54.628995 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:54.629040 (+ 45us) server_negotiation.cc:407] Connection header received
0925 04:38:54.629227 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:54.629230 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:54.629296 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:54.629403 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:38:54.629632 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:54.630164 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:54.631096 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:54.631314 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:54.631399 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:54.631686 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:38:54.632143 (+ 457us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:38:54.632247 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:54.632316 (+ 69us) server_negotiation.cc:299] Negotiation successful
0925 04:38:54.632379 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 04:38:54.644552 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:54.640473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47991 (local address 127.0.98.2:32931)
0925 04:38:54.640615 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:54.640619 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:54.640828 (+ 209us) server_negotiation.cc:407] Connection header received
0925 04:38:54.641046 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:54.641049 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:54.641100 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:54.641174 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:38:54.641467 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:54.642111 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:54.643327 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:54.643547 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:54.643677 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:54.644035 (+ 358us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:38:54.644115 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:38:54.644168 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:54.644381 (+ 213us) server_negotiation.cc:299] Negotiation successful
0925 04:38:54.644438 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 04:38:54.644622 2079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:54.640381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32931 (local address 127.0.98.1:47991)
0925 04:38:54.640736 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:54.640752 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:38:54.640891 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:38:54.641211 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:38:54.641214 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:38:54.641222 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 04:38:54.641336 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:54.641341 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:54.642220 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:54.642224 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:54.643202 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:54.643215 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:54.643708 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:54.643712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:54.643814 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:54.643873 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 04:38:54.644226 (+ 353us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 04:38:54.644236 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 04:38:54.644321 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 04:38:54.644417 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":141,"threads_started":1}
I20260925 04:38:54.651692 1933 raft_consensus.cc:1261] T e8b33cdd11ba431cbcde2d9ca97700df P 04a6633bea0742fe8224838b7e84777b [term 1 FOLLOWER]: Refusing update from remote peer e460d968d94246c18d3569e50e4b6c5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:38:54.651786 1808 raft_consensus.cc:1261] T e8b33cdd11ba431cbcde2d9ca97700df P d97dfed0caf14d53b8a09d9b161c1e9a [term 1 FOLLOWER]: Refusing update from remote peer e460d968d94246c18d3569e50e4b6c5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:38:54.652201 2048 consensus_queue.cc:1035] T e8b33cdd11ba431cbcde2d9ca97700df P e460d968d94246c18d3569e50e4b6c5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "04a6633bea0742fe8224838b7e84777b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:38:54.652403 2023 consensus_queue.cc:1035] T e8b33cdd11ba431cbcde2d9ca97700df P e460d968d94246c18d3569e50e4b6c5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d97dfed0caf14d53b8a09d9b161c1e9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:38:54.676853 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:54.672523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56885 (local address 127.0.98.1:43563)
0925 04:38:54.672648 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:54.672652 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:54.672693 (+ 41us) server_negotiation.cc:407] Connection header received
0925 04:38:54.672900 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:54.672902 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:54.672950 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:54.673031 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:38:54.673476 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:54.674298 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:54.675559 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:54.675866 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:54.675954 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:54.676340 (+ 386us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:38:54.676441 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:38:54.676517 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:54.676633 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 04:38:54.676699 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 04:38:54.676855 2085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:54.672235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43563 (local address 127.0.98.2:56885)
0925 04:38:54.672615 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:54.672632 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:38:54.672757 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:38:54.673200 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:38:54.673203 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:38:54.673213 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 04:38:54.673327 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:54.673334 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:54.674437 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:54.674440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:54.675419 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:54.675432 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:54.675991 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:54.675995 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:54.676082 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:54.676156 (+ 74us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 04:38:54.676539 (+ 383us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 04:38:54.676548 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 04:38:54.676613 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 04:38:54.676712 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":110,"threads_started":1}
I20260925 04:38:54.707441 1511 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37338:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:38:54.707667 1511 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37338:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:38:54.710227 1511 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 118324c32df744868a1da669fdde7718: Sending DeleteTablet for 3 replicas of tablet e8b33cdd11ba431cbcde2d9ca97700df
I20260925 04:38:54.711169 1648 tablet_service.cc:1514] Processing DeleteTablet for tablet e8b33cdd11ba431cbcde2d9ca97700df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:38:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41612
I20260925 04:38:54.711185 1788 tablet_service.cc:1514] Processing DeleteTablet for tablet e8b33cdd11ba431cbcde2d9ca97700df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:38:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39310
I20260925 04:38:54.711833 2089 tablet_replica.cc:330] T e8b33cdd11ba431cbcde2d9ca97700df P e460d968d94246c18d3569e50e4b6c5e: stopping tablet replica
I20260925 04:38:54.711935 1928 tablet_service.cc:1514] Processing DeleteTablet for tablet e8b33cdd11ba431cbcde2d9ca97700df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:38:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37486
I20260925 04:38:54.712218 2088 tablet_replica.cc:330] T e8b33cdd11ba431cbcde2d9ca97700df P d97dfed0caf14d53b8a09d9b161c1e9a: stopping tablet replica
I20260925 04:38:54.712405 2088 raft_consensus.cc:2229] T e8b33cdd11ba431cbcde2d9ca97700df P d97dfed0caf14d53b8a09d9b161c1e9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:38:54.712628 2089 raft_consensus.cc:2229] T e8b33cdd11ba431cbcde2d9ca97700df P e460d968d94246c18d3569e50e4b6c5e [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:38:54.712962 2088 raft_consensus.cc:2258] T e8b33cdd11ba431cbcde2d9ca97700df P d97dfed0caf14d53b8a09d9b161c1e9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:38:54.713372 2089 raft_consensus.cc:2258] T e8b33cdd11ba431cbcde2d9ca97700df P e460d968d94246c18d3569e50e4b6c5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:38:54.713692 2090 tablet_replica.cc:330] T e8b33cdd11ba431cbcde2d9ca97700df P 04a6633bea0742fe8224838b7e84777b: stopping tablet replica
I20260925 04:38:54.713840 2090 raft_consensus.cc:2229] T e8b33cdd11ba431cbcde2d9ca97700df P 04a6633bea0742fe8224838b7e84777b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:38:54.713961 2090 raft_consensus.cc:2258] T e8b33cdd11ba431cbcde2d9ca97700df P 04a6633bea0742fe8224838b7e84777b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:38:54.714529 2088 ts_tablet_manager.cc:1905] T e8b33cdd11ba431cbcde2d9ca97700df P d97dfed0caf14d53b8a09d9b161c1e9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:38:54.714671 2090 ts_tablet_manager.cc:1905] T e8b33cdd11ba431cbcde2d9ca97700df P 04a6633bea0742fe8224838b7e84777b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:38:54.715382 2089 ts_tablet_manager.cc:1905] T e8b33cdd11ba431cbcde2d9ca97700df P e460d968d94246c18d3569e50e4b6c5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:38:54.717861 2090 ts_tablet_manager.cc:1918] T e8b33cdd11ba431cbcde2d9ca97700df P 04a6633bea0742fe8224838b7e84777b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 04:38:54.717962 2090 log.cc:1196] T e8b33cdd11ba431cbcde2d9ca97700df P 04a6633bea0742fe8224838b7e84777b: Deleting WAL directory at /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/ts-2/wal/wals/e8b33cdd11ba431cbcde2d9ca97700df
I20260925 04:38:54.718209 2090 ts_tablet_manager.cc:1939] T e8b33cdd11ba431cbcde2d9ca97700df P 04a6633bea0742fe8224838b7e84777b: Deleting consensus metadata
I20260925 04:38:54.719062 1495 catalog_manager.cc:4841] TS 04a6633bea0742fe8224838b7e84777b (127.0.98.3:34353): tablet e8b33cdd11ba431cbcde2d9ca97700df (table test-table [id=38c427f7766e4477949ea8c49318d744]) successfully deleted
I20260925 04:38:54.721581 2089 ts_tablet_manager.cc:1918] T e8b33cdd11ba431cbcde2d9ca97700df P e460d968d94246c18d3569e50e4b6c5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 04:38:54.722704 2088 ts_tablet_manager.cc:1918] T e8b33cdd11ba431cbcde2d9ca97700df P d97dfed0caf14d53b8a09d9b161c1e9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 04:38:54.722887 2089 log.cc:1196] T e8b33cdd11ba431cbcde2d9ca97700df P e460d968d94246c18d3569e50e4b6c5e: Deleting WAL directory at /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/ts-0/wal/wals/e8b33cdd11ba431cbcde2d9ca97700df
I20260925 04:38:54.723011 2088 log.cc:1196] T e8b33cdd11ba431cbcde2d9ca97700df P d97dfed0caf14d53b8a09d9b161c1e9a: Deleting WAL directory at /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311123478858-392-0/minicluster-data/ts-1/wal/wals/e8b33cdd11ba431cbcde2d9ca97700df
I20260925 04:38:54.723510 2088 ts_tablet_manager.cc:1939] T e8b33cdd11ba431cbcde2d9ca97700df P d97dfed0caf14d53b8a09d9b161c1e9a: Deleting consensus metadata
I20260925 04:38:54.723898 2089 ts_tablet_manager.cc:1939] T e8b33cdd11ba431cbcde2d9ca97700df P e460d968d94246c18d3569e50e4b6c5e: Deleting consensus metadata
I20260925 04:38:54.724840 1496 catalog_manager.cc:4841] TS e460d968d94246c18d3569e50e4b6c5e (127.0.98.1:43563): tablet e8b33cdd11ba431cbcde2d9ca97700df (table test-table [id=38c427f7766e4477949ea8c49318d744]) successfully deleted
I20260925 04:38:54.725844 1498 catalog_manager.cc:4841] TS d97dfed0caf14d53b8a09d9b161c1e9a (127.0.98.2:32931): tablet e8b33cdd11ba431cbcde2d9ca97700df (table test-table [id=38c427f7766e4477949ea8c49318d744]) successfully deleted
Sep 25 04:38:54 dist-test-slave-dc95 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311133, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 04:38:54.732957 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:54.719658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41634 (local address 127.0.98.1:43563)
0925 04:38:54.719780 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:54.719785 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:54.720437 (+ 652us) server_negotiation.cc:407] Connection header received
0925 04:38:54.720634 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:54.720638 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:54.720685 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:54.720734 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:54.721646 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:54.722426 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:54.723687 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:54.724033 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:54.724103 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:54.728160 (+ 4057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:38:54.728176 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:38:54.728195 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:38:54.728231 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:38:54.730392 (+ 2161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:54.731056 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:54.731063 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:54.731065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:54.731172 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:54.731450 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:54.731453 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:54.731455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:54.731918 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:38:54.732153 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:54.732451 (+ 298us) server_negotiation.cc:299] Negotiation successful
0925 04:38:54.732756 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260925 04:38:54.733595 1708 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41634
I20260925 04:38:54.745587 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:54.735549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37356 (local address 127.0.98.62:46757)
0925 04:38:54.735987 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:54.735992 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:54.736138 (+ 146us) server_negotiation.cc:407] Connection header received
0925 04:38:54.736274 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:54.736277 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:54.736332 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:54.736412 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:54.737399 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:54.738230 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:54.739463 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:54.739858 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:54.739932 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:54.741908 (+ 1976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:38:54.741913 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:38:54.741916 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:38:54.741949 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:38:54.743678 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:54.744201 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:54.744205 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:54.744206 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:54.744303 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:54.744599 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:54.744602 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:54.744604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:54.744826 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:38:54.744928 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:54.745289 (+ 361us) server_negotiation.cc:299] Negotiation successful
0925 04:38:54.745415 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341}
W20260925 04:38:54.746760 1511 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37356
I20260925 04:38:54.748541 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 1583
I20260925 04:38:54.754369 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 1723
I20260925 04:38:54.760850 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 1863
I20260925 04:38:54.766247 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 1479
2026-09-25T04:38:54Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4406 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:38:54 dist-test-slave-dc95 krb5kdc[2097](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:38:54 dist-test-slave-dc95 krb5kdc[2097](info): set up 2 sockets
Sep 25 04:38:54 dist-test-slave-dc95 krb5kdc[2097](info): commencing operation
krb5kdc: starting...
W20260925 04:38:55.727355 392 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/krb5kdc/test-user.keytab.
Sep 25 04:38:55 dist-test-slave-dc95 krb5kdc[2097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311135, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:38:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:38:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 04:38:55.851846 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38605
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38605
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:38:55.963555 2110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:38:55.963886 2110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:38:55.963943 2110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:38:55.968091 2110 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:38:55.968158 2110 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 04:38:55.968190 2110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:38:55.968214 2110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:38:55.968236 2110 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:38:55.968266 2110 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:38:55.973790 2110 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38605
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2110
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:38:55.974905 2110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:38:55.976025 2110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:38:55.984760 2115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:38:55.984848 2118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:38:55.984818 2110 server_base.cc:1034] running on GCE node
W20260925 04:38:55.984755 2116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:38:55.985574 2110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:38:55.986884 2110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:38:55.988077 2110 hybrid_clock.cc:648] HybridClock initialized: now 1790311135988059 us; error 34 us; skew 500 ppm
Sep 25 04:38:55 dist-test-slave-dc95 krb5kdc[2097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311135, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:38:55.995198 2110 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 04:38:55.996340 2110 webserver.cc:460] Webserver started at http://127.0.98.62:41521/ using document root <none> and password file <none>
I20260925 04:38:55.996963 2110 fs_manager.cc:359] Metadata directory not provided
I20260925 04:38:55.997040 2110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:38:55.997248 2110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:38:55.999545 2110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/master-0/data/instance:
uuid: "3d4badfd4e7f4245a022ad52c0385cce"
format_stamp: "Formatted at 2026-09-25 04:38:55 on dist-test-slave-dc95"
server_key: "7a5a90aea612efbf5020ca3ad811b65d"
server_key_iv: "43682fbfccfdcf6dad93fc57f906574b"
server_key_version: "encryptionkey@0"
I20260925 04:38:56.000173 2110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/master-0/wal/instance:
uuid: "3d4badfd4e7f4245a022ad52c0385cce"
format_stamp: "Formatted at 2026-09-25 04:38:55 on dist-test-slave-dc95"
server_key: "7a5a90aea612efbf5020ca3ad811b65d"
server_key_iv: "43682fbfccfdcf6dad93fc57f906574b"
server_key_version: "encryptionkey@0"
I20260925 04:38:56.004148 2110 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:38:56.007009 2125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:38:56.008740 2110 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:38:56.008893 2110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/master-0/wal
uuid: "3d4badfd4e7f4245a022ad52c0385cce"
format_stamp: "Formatted at 2026-09-25 04:38:55 on dist-test-slave-dc95"
server_key: "7a5a90aea612efbf5020ca3ad811b65d"
server_key_iv: "43682fbfccfdcf6dad93fc57f906574b"
server_key_version: "encryptionkey@0"
I20260925 04:38:56.009024 2110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:38:56.035502 2110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:38:56.040591 2110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:38:56.040805 2110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:38:56.048750 2110 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38605
I20260925 04:38:56.048775 2187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38605 every 8 connection(s)
I20260925 04:38:56.049760 2110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
I20260925 04:38:56.052806 2188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:38:56.058728 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 2110
I20260925 04:38:56.058794 2188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d4badfd4e7f4245a022ad52c0385cce: Bootstrap starting.
I20260925 04:38:56.058840 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/master-0/wal/instance
I20260925 04:38:56.059103 392 external_mini_cluster.cc:1428] Setting key 5070ba848c38c5957a0ae010f23b9c77
I20260925 04:38:56.061532 2188 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d4badfd4e7f4245a022ad52c0385cce: Neither blocks nor log segments found. Creating new log.
I20260925 04:38:56.062301 2188 log.cc:824] T 00000000000000000000000000000000 P 3d4badfd4e7f4245a022ad52c0385cce: Log is configured to *not* fsync() on all Append() calls
I20260925 04:38:56.064322 2188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d4badfd4e7f4245a022ad52c0385cce: No bootstrap required, opened a new log
Sep 25 04:38:56 dist-test-slave-dc95 krb5kdc[2097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311135, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:38:56.068981 2188 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d4badfd4e7f4245a022ad52c0385cce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d4badfd4e7f4245a022ad52c0385cce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38605 } }
I20260925 04:38:56.069311 2188 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d4badfd4e7f4245a022ad52c0385cce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:38:56.069381 2188 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d4badfd4e7f4245a022ad52c0385cce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d4badfd4e7f4245a022ad52c0385cce, State: Initialized, Role: FOLLOWER
I20260925 04:38:56.069927 2188 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d4badfd4e7f4245a022ad52c0385cce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d4badfd4e7f4245a022ad52c0385cce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38605 } }
I20260925 04:38:56.070106 2188 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d4badfd4e7f4245a022ad52c0385cce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:38:56.070201 2188 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d4badfd4e7f4245a022ad52c0385cce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:38:56.070308 2188 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d4badfd4e7f4245a022ad52c0385cce [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:38:56.071568 2188 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d4badfd4e7f4245a022ad52c0385cce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d4badfd4e7f4245a022ad52c0385cce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38605 } }
I20260925 04:38:56.071964 2188 leader_election.cc:302] T 00000000000000000000000000000000 P 3d4badfd4e7f4245a022ad52c0385cce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d4badfd4e7f4245a022ad52c0385cce; no voters:
I20260925 04:38:56.072351 2188 leader_election.cc:288] T 00000000000000000000000000000000 P 3d4badfd4e7f4245a022ad52c0385cce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:38:56.072455 2193 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d4badfd4e7f4245a022ad52c0385cce [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:38:56.072683 2193 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d4badfd4e7f4245a022ad52c0385cce [term 1 LEADER]: Becoming Leader. State: Replica: 3d4badfd4e7f4245a022ad52c0385cce, State: Running, Role: LEADER
I20260925 04:38:56.073096 2193 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d4badfd4e7f4245a022ad52c0385cce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d4badfd4e7f4245a022ad52c0385cce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38605 } }
I20260925 04:38:56.073671 2188 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d4badfd4e7f4245a022ad52c0385cce [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:38:56.075062 2191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:56.060590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37964 (local address 127.0.98.62:38605)
0925 04:38:56.061074 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:56.061090 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:56.061125 (+ 35us) server_negotiation.cc:407] Connection header received
0925 04:38:56.061835 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:56.061858 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:56.062220 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:56.062558 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:56.063333 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:56.064372 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:56.065693 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:56.066264 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:56.066406 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:56.069282 (+ 2876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:38:56.069290 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:38:56.069311 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:38:56.069354 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:38:56.071488 (+ 2134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:56.072027 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:56.072033 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:56.072040 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:56.072190 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:56.072508 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:56.072511 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:56.072513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:56.073111 (+ 598us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:38:56.073320 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:56.073651 (+ 331us) server_negotiation.cc:299] Negotiation successful
0925 04:38:56.073985 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":143,"threads_started":1}
I20260925 04:38:56.075189 2195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d4badfd4e7f4245a022ad52c0385cce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d4badfd4e7f4245a022ad52c0385cce. Latest consensus state: current_term: 1 leader_uuid: "3d4badfd4e7f4245a022ad52c0385cce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d4badfd4e7f4245a022ad52c0385cce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38605 } } }
I20260925 04:38:56.075273 2195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d4badfd4e7f4245a022ad52c0385cce [sys.catalog]: This master's current role is: LEADER
I20260925 04:38:56.075673 2206 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:38:56.075006 2194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d4badfd4e7f4245a022ad52c0385cce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d4badfd4e7f4245a022ad52c0385cce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d4badfd4e7f4245a022ad52c0385cce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38605 } } }
I20260925 04:38:56.076660 2194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d4badfd4e7f4245a022ad52c0385cce [sys.catalog]: This master's current role is: LEADER
I20260925 04:38:56.079296 2206 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:38:56.085500 2206 catalog_manager.cc:1347] Generated new cluster ID: 67f04c690ff34da587ceeec7955a0463
I20260925 04:38:56.085582 2206 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:38:56.116037 2206 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:38:56.117054 2206 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:38:56.126449 2206 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d4badfd4e7f4245a022ad52c0385cce: Generated new TSK 0
I20260925 04:38:56.127228 2206 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 04:38:56.160053 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38605
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:38:56.275511 2214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:38:56.275836 2214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:38:56.275893 2214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:38:56.280234 2214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:38:56.280326 2214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:38:56.280368 2214 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 04:38:56.280382 2214 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 04:38:56.280417 2214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 04:38:56.286131 2214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2214
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:38:56.287396 2214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:38:56.288560 2214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:38:56.299702 2219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:38:56.299712 2220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:38:56.299770 2214 server_base.cc:1034] running on GCE node
W20260925 04:38:56.299712 2222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:38:56.300400 2214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:38:56.301079 2214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:38:56.302301 2214 hybrid_clock.cc:648] HybridClock initialized: now 1790311136302280 us; error 34 us; skew 500 ppm
Sep 25 04:38:56 dist-test-slave-dc95 krb5kdc[2097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311136, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:38:56.305301 2214 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 04:38:56.306257 2214 webserver.cc:460] Webserver started at http://127.0.98.1:38253/ using document root <none> and password file <none>
I20260925 04:38:56.306857 2214 fs_manager.cc:359] Metadata directory not provided
I20260925 04:38:56.306929 2214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:38:56.307132 2214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:38:56.309275 2214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/ts-0/data/instance:
uuid: "fc188c63638c472e899fcfa41e747aad"
format_stamp: "Formatted at 2026-09-25 04:38:56 on dist-test-slave-dc95"
server_key: "39a6e5850ba897311e6be0b40bf32f08"
server_key_iv: "ce9d8a81c42734e64c0394c6df7e9905"
server_key_version: "encryptionkey@0"
I20260925 04:38:56.309842 2214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/ts-0/wal/instance:
uuid: "fc188c63638c472e899fcfa41e747aad"
format_stamp: "Formatted at 2026-09-25 04:38:56 on dist-test-slave-dc95"
server_key: "39a6e5850ba897311e6be0b40bf32f08"
server_key_iv: "ce9d8a81c42734e64c0394c6df7e9905"
server_key_version: "encryptionkey@0"
I20260925 04:38:56.313906 2214 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 04:38:56.316738 2229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:38:56.318161 2214 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:38:56.318286 2214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/ts-0/wal
uuid: "fc188c63638c472e899fcfa41e747aad"
format_stamp: "Formatted at 2026-09-25 04:38:56 on dist-test-slave-dc95"
server_key: "39a6e5850ba897311e6be0b40bf32f08"
server_key_iv: "ce9d8a81c42734e64c0394c6df7e9905"
server_key_version: "encryptionkey@0"
I20260925 04:38:56.318480 2214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:38:56.332512 2214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:38:56.336028 2214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:38:56.336223 2214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:38:56.337714 2214 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:38:56.337770 2214 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:38:56.337823 2214 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:38:56.337852 2214 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:38:56.347352 2214 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41049
I20260925 04:38:56.347436 2344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41049 every 8 connection(s)
I20260925 04:38:56.348515 2214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
Sep 25 04:38:56 dist-test-slave-dc95 krb5kdc[2097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311136, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:38:56.353807 2191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:56.338651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49861 (local address 127.0.98.62:38605)
0925 04:38:56.338804 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:56.338809 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:56.339738 (+ 929us) server_negotiation.cc:407] Connection header received
0925 04:38:56.341012 (+ 1274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:56.341016 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:56.341085 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:56.341191 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:56.343299 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:56.344178 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:56.346121 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:56.346564 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:56.346683 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:56.349834 (+ 3151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:38:56.349837 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:38:56.349839 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:38:56.349868 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:38:56.351354 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:56.352024 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:56.352028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:56.352029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:56.352128 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:56.352564 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:56.352567 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:56.352569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:56.352786 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:38:56.352900 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:56.353551 (+ 651us) server_negotiation.cc:299] Negotiation successful
0925 04:38:56.353677 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 04:38:56.354646 2238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:56.339060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38605 (local address 127.0.98.1:49861)
0925 04:38:56.339548 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:56.339600 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 04:38:56.340727 (+ 1127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:38:56.341541 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:38:56.341551 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:38:56.342200 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:38:56.343036 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:56.343063 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:56.344386 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:56.344391 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:56.345958 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:56.345975 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:56.346759 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:56.346763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:56.346960 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:56.347759 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:38:56.347839 (+ 80us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:38:56.349612 (+ 1773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:38:56.351509 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:56.351515 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:56.351530 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:56.351897 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:56.352244 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:56.352247 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:56.352248 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:56.352430 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:56.352931 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:38:56.352937 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:38:56.353286 (+ 349us) client_negotiation.cc:769] Sending connection context
0925 04:38:56.353465 (+ 179us) client_negotiation.cc:240] Negotiation successful
0925 04:38:56.353756 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":126,"threads_started":1}
I20260925 04:38:56.356202 2345 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38605
I20260925 04:38:56.356391 2345 heartbeater.cc:464] Registering TS with master...
I20260925 04:38:56.356989 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 2214
I20260925 04:38:56.357034 2345 heartbeater.cc:511] Master 127.0.98.62:38605 requested a full tablet report, sending...
I20260925 04:38:56.357100 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/ts-0/wal/instance
I20260925 04:38:56.357401 392 external_mini_cluster.cc:1428] Setting key 138ccfaf2182bd1b3441ca9e21d90522
I20260925 04:38:56.358886 2142 ts_manager.cc:194] Registered new tserver with Master: fc188c63638c472e899fcfa41e747aad (127.0.98.1:41049)
I20260925 04:38:56.360476 2142 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49861
I20260925 04:38:56.366245 2351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:56.358510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38605 (local address 127.0.98.1:42345)
0925 04:38:56.358896 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:56.358908 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:38:56.359006 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:38:56.359244 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:38:56.359247 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:38:56.359452 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:38:56.359556 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:56.359561 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:56.360486 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:56.360489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:56.361126 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:56.361137 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:56.361716 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:56.361720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:56.361812 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:56.362356 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:38:56.362371 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:38:56.362803 (+ 432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:38:56.364632 (+ 1829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:56.364634 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:56.364636 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:56.364842 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:56.365187 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:56.365190 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:56.365192 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:56.365291 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:56.365744 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:38:56.365746 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:38:56.365857 (+ 111us) client_negotiation.cc:769] Sending connection context
0925 04:38:56.365958 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 04:38:56.366082 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":114,"threads_started":1}
I20260925 04:38:56.366317 2191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:56.358596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42345 (local address 127.0.98.62:38605)
0925 04:38:56.358710 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:56.358714 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:56.358979 (+ 265us) server_negotiation.cc:407] Connection header received
0925 04:38:56.359082 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:56.359086 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:56.359131 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:56.359201 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:56.359686 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:56.360309 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:56.361252 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:56.361576 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:56.361682 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:56.362973 (+ 1291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:38:56.362977 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:38:56.362980 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:38:56.363006 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:38:56.364521 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:56.364974 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:56.364978 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:56.364980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:56.365067 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:56.365388 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:56.365391 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:56.365393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:56.365617 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:38:56.365729 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:56.366033 (+ 304us) server_negotiation.cc:299] Negotiation successful
0925 04:38:56.366152 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":83,"server-negotiator.queue_time_us":43}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 04:38:56.387403 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38605
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:38:56.499357 2354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:38:56.499687 2354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:38:56.499747 2354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:38:56.503955 2354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:38:56.504025 2354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:38:56.504068 2354 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 04:38:56.504096 2354 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 04:38:56.504146 2354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 04:38:56.509722 2354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:38:56.510846 2354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:38:56.511929 2354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:38:56.522504 2359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:38:56.522504 2360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:38:56.522535 2362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:38:56.522976 2354 server_base.cc:1034] running on GCE node
I20260925 04:38:56.523558 2354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:38:56.524338 2354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:38:56.525542 2354 hybrid_clock.cc:648] HybridClock initialized: now 1790311136525525 us; error 33 us; skew 500 ppm
Sep 25 04:38:56 dist-test-slave-dc95 krb5kdc[2097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311136, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:38:56.528679 2354 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 04:38:56.529654 2354 webserver.cc:460] Webserver started at http://127.0.98.2:38761/ using document root <none> and password file <none>
I20260925 04:38:56.530241 2354 fs_manager.cc:359] Metadata directory not provided
I20260925 04:38:56.530372 2354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:38:56.530587 2354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:38:56.532828 2354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/ts-1/data/instance:
uuid: "90534941bd5549558e508bdb29ac8265"
format_stamp: "Formatted at 2026-09-25 04:38:56 on dist-test-slave-dc95"
server_key: "fe70b1852a24408bcabe79b821ccb147"
server_key_iv: "0762e02d9a56b39411c7ae3c11cac64d"
server_key_version: "encryptionkey@0"
I20260925 04:38:56.533388 2354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/ts-1/wal/instance:
uuid: "90534941bd5549558e508bdb29ac8265"
format_stamp: "Formatted at 2026-09-25 04:38:56 on dist-test-slave-dc95"
server_key: "fe70b1852a24408bcabe79b821ccb147"
server_key_iv: "0762e02d9a56b39411c7ae3c11cac64d"
server_key_version: "encryptionkey@0"
I20260925 04:38:56.537461 2354 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 04:38:56.540148 2369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:38:56.541621 2354 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:38:56.541751 2354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/ts-1/wal
uuid: "90534941bd5549558e508bdb29ac8265"
format_stamp: "Formatted at 2026-09-25 04:38:56 on dist-test-slave-dc95"
server_key: "fe70b1852a24408bcabe79b821ccb147"
server_key_iv: "0762e02d9a56b39411c7ae3c11cac64d"
server_key_version: "encryptionkey@0"
I20260925 04:38:56.541869 2354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:38:56.565652 2354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:38:56.569381 2354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:38:56.569573 2354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:38:56.571076 2354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:38:56.571130 2354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:38:56.571171 2354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:38:56.571198 2354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 04:38:56 dist-test-slave-dc95 krb5kdc[2097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311136, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:38:56.581652 2354 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41491
I20260925 04:38:56.581861 2484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41491 every 8 connection(s)
I20260925 04:38:56.582625 2354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
I20260925 04:38:56.584453 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 2354
I20260925 04:38:56.584556 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/ts-1/wal/instance
I20260925 04:38:56.584856 392 external_mini_cluster.cc:1428] Setting key d45a9baf000e6aa1e09453920be69b6d
I20260925 04:38:56.585747 2191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:56.572165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47959 (local address 127.0.98.62:38605)
0925 04:38:56.572308 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:56.572313 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:56.573478 (+ 1165us) server_negotiation.cc:407] Connection header received
0925 04:38:56.574822 (+ 1344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:56.574826 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:56.574887 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:56.574964 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:56.576786 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:56.577600 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:56.578843 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:56.579065 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:56.579158 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:56.581829 (+ 2671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:38:56.581833 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:38:56.581836 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:38:56.581874 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:38:56.583320 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:56.583965 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:56.583968 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:56.583969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:56.584061 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:56.584547 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:56.584551 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:56.584553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:56.584806 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:38:56.584910 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:56.585472 (+ 562us) server_negotiation.cc:299] Negotiation successful
0925 04:38:56.585597 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 04:38:56.586659 2378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:56.572720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38605 (local address 127.0.98.2:47959)
0925 04:38:56.573297 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:56.573344 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 04:38:56.574497 (+ 1153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:38:56.575232 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:38:56.575242 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:38:56.575827 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:38:56.576589 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:56.576611 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:56.577767 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:56.577771 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:56.578713 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:56.578723 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:56.579216 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:56.579221 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:56.579409 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:56.580066 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:38:56.580093 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:38:56.581616 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:38:56.583464 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:56.583470 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:56.583484 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:56.583810 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:56.584186 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:56.584189 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:56.584190 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:56.584422 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:56.584959 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:38:56.584967 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:38:56.585225 (+ 258us) client_negotiation.cc:769] Sending connection context
0925 04:38:56.585418 (+ 193us) client_negotiation.cc:240] Negotiation successful
0925 04:38:56.585725 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":157,"threads_started":1}
I20260925 04:38:56.588024 2485 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38605
I20260925 04:38:56.588181 2485 heartbeater.cc:464] Registering TS with master...
I20260925 04:38:56.588792 2485 heartbeater.cc:511] Master 127.0.98.62:38605 requested a full tablet report, sending...
I20260925 04:38:56.589825 2141 ts_manager.cc:194] Registered new tserver with Master: 90534941bd5549558e508bdb29ac8265 (127.0.98.2:41491)
I20260925 04:38:56.590534 2141 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47959
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260925 04:38:56.598255 2492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:56.590123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38605 (local address 127.0.98.2:33831)
0925 04:38:56.590914 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:56.590929 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:38:56.591023 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:38:56.591273 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:38:56.591276 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:38:56.591491 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:38:56.591632 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:56.591640 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:56.592641 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:56.592644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:56.593199 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:56.593207 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:56.593800 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:56.593804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:56.593889 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:56.594417 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:38:56.594428 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:38:56.594765 (+ 337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:38:56.596533 (+ 1768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:56.596537 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:56.596538 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:56.596794 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:56.597142 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:56.597145 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:56.597148 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:56.597253 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:56.597678 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:38:56.597682 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:38:56.597782 (+ 100us) client_negotiation.cc:769] Sending connection context
0925 04:38:56.597874 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 04:38:56.598051 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"thread_start_us":102,"threads_started":1}
I20260925 04:38:56.598250 2191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:56.590203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33831 (local address 127.0.98.62:38605)
0925 04:38:56.590325 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:56.590329 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:56.591002 (+ 673us) server_negotiation.cc:407] Connection header received
0925 04:38:56.591102 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:56.591105 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:56.591155 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:56.591240 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:56.591735 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:56.592514 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:56.593326 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:56.593659 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:56.593765 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:56.594974 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:38:56.594986 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:38:56.594989 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:38:56.595021 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:38:56.596417 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:56.596935 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:56.596938 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:56.596939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:56.597024 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:56.597328 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:56.597331 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:56.597332 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:56.597584 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:38:56.597677 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:56.597959 (+ 282us) server_negotiation.cc:299] Negotiation successful
0925 04:38:56.598076 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 04:38:56.615301 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38605
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:38:56.729254 2494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:38:56.729580 2494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:38:56.729630 2494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:38:56.734242 2494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:38:56.734340 2494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:38:56.734385 2494 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 04:38:56.734398 2494 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 04:38:56.734445 2494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 04:38:56.740231 2494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:38:56.741417 2494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:38:56.742460 2494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:38:56.752511 2500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:38:56.752676 2494 server_base.cc:1034] running on GCE node
W20260925 04:38:56.752504 2502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:38:56.752511 2499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:38:56.753170 2494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:38:56.753836 2494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:38:56.755020 2494 hybrid_clock.cc:648] HybridClock initialized: now 1790311136755006 us; error 30 us; skew 500 ppm
Sep 25 04:38:56 dist-test-slave-dc95 krb5kdc[2097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311136, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:38:56.758072 2494 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 04:38:56.759065 2494 webserver.cc:460] Webserver started at http://127.0.98.3:34461/ using document root <none> and password file <none>
I20260925 04:38:56.759658 2494 fs_manager.cc:359] Metadata directory not provided
I20260925 04:38:56.759729 2494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:38:56.759938 2494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:38:56.762028 2494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/ts-2/data/instance:
uuid: "41c1bfb87491488db8b4877520e5ac60"
format_stamp: "Formatted at 2026-09-25 04:38:56 on dist-test-slave-dc95"
server_key: "4069b48050987348937a87179f60dcfc"
server_key_iv: "58d628303c3dd18a7083a44090405bf4"
server_key_version: "encryptionkey@0"
I20260925 04:38:56.762563 2494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/ts-2/wal/instance:
uuid: "41c1bfb87491488db8b4877520e5ac60"
format_stamp: "Formatted at 2026-09-25 04:38:56 on dist-test-slave-dc95"
server_key: "4069b48050987348937a87179f60dcfc"
server_key_iv: "58d628303c3dd18a7083a44090405bf4"
server_key_version: "encryptionkey@0"
I20260925 04:38:56.766247 2494 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260925 04:38:56.768810 2509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:38:56.770201 2494 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 04:38:56.770352 2494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/ts-2/wal
uuid: "41c1bfb87491488db8b4877520e5ac60"
format_stamp: "Formatted at 2026-09-25 04:38:56 on dist-test-slave-dc95"
server_key: "4069b48050987348937a87179f60dcfc"
server_key_iv: "58d628303c3dd18a7083a44090405bf4"
server_key_version: "encryptionkey@0"
I20260925 04:38:56.770458 2494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:38:56.795383 2494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:38:56.798902 2494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:38:56.799118 2494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:38:56.800755 2494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:38:56.800815 2494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:38:56.800869 2494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:38:56.800892 2494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:38:56.811252 2494 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38467
I20260925 04:38:56.811293 2624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38467 every 8 connection(s)
Sep 25 04:38:56 dist-test-slave-dc95 krb5kdc[2097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311136, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:38:56.812273 2494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
I20260925 04:38:56.816033 2191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:56.801831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34169 (local address 127.0.98.62:38605)
0925 04:38:56.801954 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:56.801957 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:56.803104 (+ 1147us) server_negotiation.cc:407] Connection header received
0925 04:38:56.804271 (+ 1167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:56.804275 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:56.804377 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:56.804462 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:56.805990 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:56.806824 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:56.808728 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:56.809026 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:56.809157 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:56.812372 (+ 3215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:38:56.812375 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:38:56.812377 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:38:56.812408 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:38:56.813891 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:56.814515 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:56.814519 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:56.814520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:56.814606 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:56.814990 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:56.814994 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:56.814995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:56.815192 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:38:56.815272 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:56.815782 (+ 510us) server_negotiation.cc:299] Negotiation successful
0925 04:38:56.815887 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 04:38:56.816895 2518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:56.802407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38605 (local address 127.0.98.3:34169)
0925 04:38:56.802939 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:56.802985 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 04:38:56.804048 (+ 1063us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:38:56.804668 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:38:56.804676 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:38:56.805111 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:38:56.805832 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:56.805856 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:56.807007 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:56.807012 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:56.808545 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:56.808562 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:56.809229 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:56.809235 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:56.809478 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:56.810210 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:38:56.810232 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:38:56.812159 (+ 1927us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:38:56.814024 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:56.814029 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:56.814041 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:56.814368 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:56.814710 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:56.814713 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:56.814715 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:56.814869 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:56.815270 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:38:56.815276 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:38:56.815519 (+ 243us) client_negotiation.cc:769] Sending connection context
0925 04:38:56.815710 (+ 191us) client_negotiation.cc:240] Negotiation successful
0925 04:38:56.816003 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":139,"threads_started":1}
I20260925 04:38:56.818145 2625 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38605
I20260925 04:38:56.818298 2625 heartbeater.cc:464] Registering TS with master...
I20260925 04:38:56.818918 2625 heartbeater.cc:511] Master 127.0.98.62:38605 requested a full tablet report, sending...
I20260925 04:38:56.820084 2142 ts_manager.cc:194] Registered new tserver with Master: 41c1bfb87491488db8b4877520e5ac60 (127.0.98.3:38467)
I20260925 04:38:56.820827 2142 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34169
I20260925 04:38:56.822196 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 2494
I20260925 04:38:56.822312 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/ts-2/wal/instance
I20260925 04:38:56.822579 392 external_mini_cluster.cc:1428] Setting key 6a439eaa7ab25962b950ad3db54af6d6
I20260925 04:38:56.824944 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:38:56.828151 2191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:56.820602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59895 (local address 127.0.98.62:38605)
0925 04:38:56.820713 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:56.820718 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:56.820866 (+ 148us) server_negotiation.cc:407] Connection header received
0925 04:38:56.821009 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:56.821012 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:56.821054 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:56.821117 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:56.821673 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:56.822197 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:56.823301 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:56.823632 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:56.823737 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:56.824965 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:38:56.824969 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:38:56.824972 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:38:56.824999 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:38:56.826521 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:56.826972 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:56.826976 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:56.826978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:56.827067 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:56.827348 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:56.827350 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:56.827351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:56.827528 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:38:56.827629 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:56.827932 (+ 303us) server_negotiation.cc:299] Negotiation successful
0925 04:38:56.828034 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 04:38:56.828228 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:56.820534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38605 (local address 127.0.98.3:59895)
0925 04:38:56.820793 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:56.820805 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:38:56.820898 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:38:56.821145 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:38:56.821148 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:38:56.821407 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:38:56.821536 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:56.821543 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:56.822442 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:56.822446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:56.823171 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:56.823182 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:56.823737 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:56.823741 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:56.823833 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:56.824448 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:38:56.824460 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:38:56.824847 (+ 387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:38:56.826642 (+ 1795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:56.826645 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:56.826646 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:56.826866 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:56.827143 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:56.827146 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:56.827148 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:56.827258 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:56.827623 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:38:56.827626 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:38:56.827743 (+ 117us) client_negotiation.cc:769] Sending connection context
0925 04:38:56.827859 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 04:38:56.828023 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":101,"threads_started":1}
Sep 25 04:38:56 dist-test-slave-dc95 krb5kdc[2097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 04:38:56 dist-test-slave-dc95 krb5kdc[2097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:38:56.851759 2191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:56.843451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37978 (local address 127.0.98.62:38605)
0925 04:38:56.843587 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:56.843590 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:56.843635 (+ 45us) server_negotiation.cc:407] Connection header received
0925 04:38:56.843682 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:56.843684 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:56.843723 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:56.843793 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:56.844497 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:56.845033 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:56.845925 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:56.846137 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:56.846225 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:56.848700 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:38:56.848703 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:38:56.848705 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:38:56.848732 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:38:56.850126 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:56.850552 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:56.850555 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:56.850557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:56.850634 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:56.850883 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:56.850886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:56.850888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:56.851073 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:38:56.851190 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:56.851447 (+ 257us) server_negotiation.cc:299] Negotiation successful
0925 04:38:56.851565 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 04:38:56.854826 2142 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37978:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:38:56.858291 2142 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:38:56.873330 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:56.867940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38467 (local address 127.0.0.1:36102)
0925 04:38:56.868668 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:56.868680 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:38:56.868763 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:38:56.869430 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:38:56.869433 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:38:56.869448 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:38:56.869543 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:56.869550 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:56.870640 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:56.870644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:56.871934 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:56.871945 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:56.872951 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:56.872955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:56.873047 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:56.873100 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 04:38:56.873162 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 04:38:56.873237 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"mutex_wait_us":43,"thread_start_us":84,"threads_started":1}
I20260925 04:38:56.873387 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:56.867888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37366 (local address 127.0.98.1:41049)
0925 04:38:56.868165 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:56.868171 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:56.868668 (+ 497us) server_negotiation.cc:407] Connection header received
0925 04:38:56.868747 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:56.868752 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:56.868940 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:56.869073 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:38:56.869257 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:56.870371 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:56.872075 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:56.872603 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:56.872704 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:56.872858 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:56.873137 (+ 279us) server_negotiation.cc:299] Negotiation successful
0925 04:38:56.873246 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":90,"threads_started":1}
I20260925 04:38:56.873298 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:56.867337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41049 (local address 127.0.0.1:37366)
0925 04:38:56.868483 (+ 1146us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:56.868497 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:38:56.868581 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:38:56.869054 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:38:56.869057 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:38:56.869076 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:38:56.869165 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:56.869170 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:56.870465 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:56.870468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:56.871934 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:56.871945 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:56.872720 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:56.872724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:56.872855 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:56.872897 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 04:38:56.873005 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 04:38:56.873133 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1087,"mutex_wait_us":103,"thread_start_us":95,"threads_started":1}
I20260925 04:38:56.873662 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:56.867131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41491 (local address 127.0.0.1:41962)
0925 04:38:56.867471 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:56.867697 (+ 226us) client_negotiation.cc:174] Beginning negotiation
0925 04:38:56.867833 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:38:56.868744 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:38:56.868750 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:38:56.868770 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:38:56.868911 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:56.868917 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:56.870429 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:56.870433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:56.871730 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:56.871741 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:56.872970 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:56.872973 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:56.873393 (+ 420us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:56.873414 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 04:38:56.873480 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 04:38:56.873545 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":357,"thread_start_us":91,"threads_started":1}
I20260925 04:38:56.873883 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:56.867301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41962 (local address 127.0.98.2:41491)
0925 04:38:56.867903 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:56.867914 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:56.867954 (+ 40us) server_negotiation.cc:407] Connection header received
0925 04:38:56.868049 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:56.868054 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:56.868274 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:56.868441 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:38:56.869046 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:56.870305 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:56.871836 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:56.872642 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:56.872747 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:56.872901 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:56.873648 (+ 747us) server_negotiation.cc:299] Negotiation successful
0925 04:38:56.873747 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":119,"threads_started":1}
I20260925 04:38:56.874058 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:56.868314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36102 (local address 127.0.98.3:38467)
0925 04:38:56.869021 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:56.869032 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:56.869052 (+ 20us) server_negotiation.cc:407] Connection header received
0925 04:38:56.869118 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:56.869123 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:56.869307 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:56.869432 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:38:56.869646 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:56.870535 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:56.872059 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:56.872815 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:56.872953 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:56.873097 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:56.873182 (+ 85us) server_negotiation.cc:299] Negotiation successful
0925 04:38:56.873255 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":111,"threads_started":1}
I20260925 04:38:56.876674 2279 tablet_service.cc:1467] Processing CreateTablet for tablet 626b1f46eca84651875ead9fd8e747f1 (DEFAULT_TABLE table=test-table [id=c51d7b60229440a6a77d1a4d8beff158]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:38:56.876780 2559 tablet_service.cc:1467] Processing CreateTablet for tablet 626b1f46eca84651875ead9fd8e747f1 (DEFAULT_TABLE table=test-table [id=c51d7b60229440a6a77d1a4d8beff158]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:38:56.876777 2419 tablet_service.cc:1467] Processing CreateTablet for tablet 626b1f46eca84651875ead9fd8e747f1 (DEFAULT_TABLE table=test-table [id=c51d7b60229440a6a77d1a4d8beff158]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:38:56.877784 2279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 626b1f46eca84651875ead9fd8e747f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:38:56.877825 2419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 626b1f46eca84651875ead9fd8e747f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:38:56.877833 2559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 626b1f46eca84651875ead9fd8e747f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:38:56.884799 2648 tablet_bootstrap.cc:492] T 626b1f46eca84651875ead9fd8e747f1 P fc188c63638c472e899fcfa41e747aad: Bootstrap starting.
I20260925 04:38:56.886250 2650 tablet_bootstrap.cc:492] T 626b1f46eca84651875ead9fd8e747f1 P 90534941bd5549558e508bdb29ac8265: Bootstrap starting.
I20260925 04:38:56.886664 2649 tablet_bootstrap.cc:492] T 626b1f46eca84651875ead9fd8e747f1 P 41c1bfb87491488db8b4877520e5ac60: Bootstrap starting.
I20260925 04:38:56.887125 2648 tablet_bootstrap.cc:654] T 626b1f46eca84651875ead9fd8e747f1 P fc188c63638c472e899fcfa41e747aad: Neither blocks nor log segments found. Creating new log.
I20260925 04:38:56.887985 2648 log.cc:824] T 626b1f46eca84651875ead9fd8e747f1 P fc188c63638c472e899fcfa41e747aad: Log is configured to *not* fsync() on all Append() calls
I20260925 04:38:56.888597 2650 tablet_bootstrap.cc:654] T 626b1f46eca84651875ead9fd8e747f1 P 90534941bd5549558e508bdb29ac8265: Neither blocks nor log segments found. Creating new log.
I20260925 04:38:56.888597 2649 tablet_bootstrap.cc:654] T 626b1f46eca84651875ead9fd8e747f1 P 41c1bfb87491488db8b4877520e5ac60: Neither blocks nor log segments found. Creating new log.
I20260925 04:38:56.889439 2650 log.cc:824] T 626b1f46eca84651875ead9fd8e747f1 P 90534941bd5549558e508bdb29ac8265: Log is configured to *not* fsync() on all Append() calls
I20260925 04:38:56.889437 2649 log.cc:824] T 626b1f46eca84651875ead9fd8e747f1 P 41c1bfb87491488db8b4877520e5ac60: Log is configured to *not* fsync() on all Append() calls
I20260925 04:38:56.889994 2648 tablet_bootstrap.cc:492] T 626b1f46eca84651875ead9fd8e747f1 P fc188c63638c472e899fcfa41e747aad: No bootstrap required, opened a new log
I20260925 04:38:56.890228 2648 ts_tablet_manager.cc:1397] T 626b1f46eca84651875ead9fd8e747f1 P fc188c63638c472e899fcfa41e747aad: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 04:38:56.891242 2649 tablet_bootstrap.cc:492] T 626b1f46eca84651875ead9fd8e747f1 P 41c1bfb87491488db8b4877520e5ac60: No bootstrap required, opened a new log
I20260925 04:38:56.891414 2649 ts_tablet_manager.cc:1397] T 626b1f46eca84651875ead9fd8e747f1 P 41c1bfb87491488db8b4877520e5ac60: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 04:38:56.891728 2650 tablet_bootstrap.cc:492] T 626b1f46eca84651875ead9fd8e747f1 P 90534941bd5549558e508bdb29ac8265: No bootstrap required, opened a new log
I20260925 04:38:56.891922 2650 ts_tablet_manager.cc:1397] T 626b1f46eca84651875ead9fd8e747f1 P 90534941bd5549558e508bdb29ac8265: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 04:38:56.894331 2649 raft_consensus.cc:348] T 626b1f46eca84651875ead9fd8e747f1 P 41c1bfb87491488db8b4877520e5ac60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90534941bd5549558e508bdb29ac8265" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "41c1bfb87491488db8b4877520e5ac60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 } } peers { permanent_uuid: "fc188c63638c472e899fcfa41e747aad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41049 } }
I20260925 04:38:56.894562 2649 raft_consensus.cc:374] T 626b1f46eca84651875ead9fd8e747f1 P 41c1bfb87491488db8b4877520e5ac60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:38:56.894637 2649 raft_consensus.cc:729] T 626b1f46eca84651875ead9fd8e747f1 P 41c1bfb87491488db8b4877520e5ac60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c1bfb87491488db8b4877520e5ac60, State: Initialized, Role: FOLLOWER
I20260925 04:38:56.894774 2648 raft_consensus.cc:348] T 626b1f46eca84651875ead9fd8e747f1 P fc188c63638c472e899fcfa41e747aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90534941bd5549558e508bdb29ac8265" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "41c1bfb87491488db8b4877520e5ac60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 } } peers { permanent_uuid: "fc188c63638c472e899fcfa41e747aad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41049 } }
I20260925 04:38:56.895052 2648 raft_consensus.cc:374] T 626b1f46eca84651875ead9fd8e747f1 P fc188c63638c472e899fcfa41e747aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:38:56.895143 2648 raft_consensus.cc:729] T 626b1f46eca84651875ead9fd8e747f1 P fc188c63638c472e899fcfa41e747aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc188c63638c472e899fcfa41e747aad, State: Initialized, Role: FOLLOWER
I20260925 04:38:56.895093 2649 consensus_queue.cc:260] T 626b1f46eca84651875ead9fd8e747f1 P 41c1bfb87491488db8b4877520e5ac60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90534941bd5549558e508bdb29ac8265" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "41c1bfb87491488db8b4877520e5ac60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 } } peers { permanent_uuid: "fc188c63638c472e899fcfa41e747aad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41049 } }
I20260925 04:38:56.895582 2648 consensus_queue.cc:260] T 626b1f46eca84651875ead9fd8e747f1 P fc188c63638c472e899fcfa41e747aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90534941bd5549558e508bdb29ac8265" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "41c1bfb87491488db8b4877520e5ac60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 } } peers { permanent_uuid: "fc188c63638c472e899fcfa41e747aad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41049 } }
I20260925 04:38:56.895802 2625 heartbeater.cc:503] Master 127.0.98.62:38605 was elected leader, sending a full tablet report...
I20260925 04:38:56.895749 2650 raft_consensus.cc:348] T 626b1f46eca84651875ead9fd8e747f1 P 90534941bd5549558e508bdb29ac8265 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90534941bd5549558e508bdb29ac8265" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "41c1bfb87491488db8b4877520e5ac60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 } } peers { permanent_uuid: "fc188c63638c472e899fcfa41e747aad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41049 } }
I20260925 04:38:56.895985 2650 raft_consensus.cc:374] T 626b1f46eca84651875ead9fd8e747f1 P 90534941bd5549558e508bdb29ac8265 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:38:56.896034 2650 raft_consensus.cc:729] T 626b1f46eca84651875ead9fd8e747f1 P 90534941bd5549558e508bdb29ac8265 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90534941bd5549558e508bdb29ac8265, State: Initialized, Role: FOLLOWER
I20260925 04:38:56.896627 2648 ts_tablet_manager.cc:1428] T 626b1f46eca84651875ead9fd8e747f1 P fc188c63638c472e899fcfa41e747aad: Time spent starting tablet: real 0.006s user 0.000s sys 0.005s
I20260925 04:38:56.896792 2650 consensus_queue.cc:260] T 626b1f46eca84651875ead9fd8e747f1 P 90534941bd5549558e508bdb29ac8265 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90534941bd5549558e508bdb29ac8265" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "41c1bfb87491488db8b4877520e5ac60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 } } peers { permanent_uuid: "fc188c63638c472e899fcfa41e747aad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41049 } }
I20260925 04:38:56.896966 2345 heartbeater.cc:503] Master 127.0.98.62:38605 was elected leader, sending a full tablet report...
I20260925 04:38:56.897815 2650 ts_tablet_manager.cc:1428] T 626b1f46eca84651875ead9fd8e747f1 P 90534941bd5549558e508bdb29ac8265: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260925 04:38:56.898022 2649 ts_tablet_manager.cc:1428] T 626b1f46eca84651875ead9fd8e747f1 P 41c1bfb87491488db8b4877520e5ac60: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260925 04:38:56.898221 2485 heartbeater.cc:503] Master 127.0.98.62:38605 was elected leader, sending a full tablet report...
I20260925 04:38:57.037531 2656 raft_consensus.cc:482] T 626b1f46eca84651875ead9fd8e747f1 P 90534941bd5549558e508bdb29ac8265 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:38:57.037753 2656 raft_consensus.cc:504] T 626b1f46eca84651875ead9fd8e747f1 P 90534941bd5549558e508bdb29ac8265 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90534941bd5549558e508bdb29ac8265" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "41c1bfb87491488db8b4877520e5ac60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 } } peers { permanent_uuid: "fc188c63638c472e899fcfa41e747aad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41049 } }
I20260925 04:38:57.038996 2656 leader_election.cc:288] T 626b1f46eca84651875ead9fd8e747f1 P 90534941bd5549558e508bdb29ac8265 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41c1bfb87491488db8b4877520e5ac60 (127.0.98.3:38467), fc188c63638c472e899fcfa41e747aad (127.0.98.1:41049)
I20260925 04:38:57.043953 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:57.039434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58365 (local address 127.0.98.1:41049)
0925 04:38:57.039576 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:57.039580 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:57.039648 (+ 68us) server_negotiation.cc:407] Connection header received
0925 04:38:57.039710 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:57.039713 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:57.039768 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:57.039846 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:38:57.040161 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:57.040962 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:57.042656 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:57.043236 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:57.043428 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:57.043499 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:57.043724 (+ 225us) server_negotiation.cc:299] Negotiation successful
0925 04:38:57.043800 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 04:38:57.043963 2378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:57.039319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41049 (local address 127.0.98.2:58365)
0925 04:38:57.039445 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:57.039460 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:38:57.039563 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:38:57.039874 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:38:57.039877 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:38:57.039900 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:38:57.040020 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:57.040026 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:57.041092 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:57.041095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:57.042469 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:57.042482 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:57.043459 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:57.043462 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:57.043563 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:57.043590 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 04:38:57.043706 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 04:38:57.043793 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 04:38:57.043999 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:57.039418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54525 (local address 127.0.98.3:38467)
0925 04:38:57.039564 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:57.039569 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:57.039961 (+ 392us) server_negotiation.cc:407] Connection header received
0925 04:38:57.040117 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:57.040121 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:57.040179 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:57.040263 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:38:57.040492 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:57.041169 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:57.042911 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:57.043472 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:57.043567 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:57.043642 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:57.043827 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 04:38:57.043900 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 04:38:57.043974 2657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:57.039321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38467 (local address 127.0.98.2:54525)
0925 04:38:57.039883 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:57.039894 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 04:38:57.039983 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:38:57.040278 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:38:57.040280 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:38:57.040297 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:38:57.040388 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:57.040393 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:57.041271 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:57.041275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:57.042787 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:57.042795 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:57.043599 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:57.043626 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:57.043716 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:57.043734 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 04:38:57.043803 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 04:38:57.043857 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"mutex_wait_us":156,"thread_start_us":184,"threads_started":1}
I20260925 04:38:57.044802 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "626b1f46eca84651875ead9fd8e747f1" candidate_uuid: "90534941bd5549558e508bdb29ac8265" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c1bfb87491488db8b4877520e5ac60" is_pre_election: true
I20260925 04:38:57.044823 2299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "626b1f46eca84651875ead9fd8e747f1" candidate_uuid: "90534941bd5549558e508bdb29ac8265" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc188c63638c472e899fcfa41e747aad" is_pre_election: true
I20260925 04:38:57.045091 2579 raft_consensus.cc:2454] T 626b1f46eca84651875ead9fd8e747f1 P 41c1bfb87491488db8b4877520e5ac60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90534941bd5549558e508bdb29ac8265 in term 0.
I20260925 04:38:57.045094 2299 raft_consensus.cc:2454] T 626b1f46eca84651875ead9fd8e747f1 P fc188c63638c472e899fcfa41e747aad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90534941bd5549558e508bdb29ac8265 in term 0.
I20260925 04:38:57.045568 2370 leader_election.cc:302] T 626b1f46eca84651875ead9fd8e747f1 P 90534941bd5549558e508bdb29ac8265 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41c1bfb87491488db8b4877520e5ac60, 90534941bd5549558e508bdb29ac8265; no voters:
I20260925 04:38:57.045840 2656 raft_consensus.cc:2781] T 626b1f46eca84651875ead9fd8e747f1 P 90534941bd5549558e508bdb29ac8265 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:38:57.045918 2656 raft_consensus.cc:482] T 626b1f46eca84651875ead9fd8e747f1 P 90534941bd5549558e508bdb29ac8265 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:38:57.045969 2656 raft_consensus.cc:3037] T 626b1f46eca84651875ead9fd8e747f1 P 90534941bd5549558e508bdb29ac8265 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:38:57.047034 2656 raft_consensus.cc:504] T 626b1f46eca84651875ead9fd8e747f1 P 90534941bd5549558e508bdb29ac8265 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90534941bd5549558e508bdb29ac8265" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "41c1bfb87491488db8b4877520e5ac60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 } } peers { permanent_uuid: "fc188c63638c472e899fcfa41e747aad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41049 } }
I20260925 04:38:57.047372 2656 leader_election.cc:288] T 626b1f46eca84651875ead9fd8e747f1 P 90534941bd5549558e508bdb29ac8265 [CANDIDATE]: Term 1 election: Requested vote from peers 41c1bfb87491488db8b4877520e5ac60 (127.0.98.3:38467), fc188c63638c472e899fcfa41e747aad (127.0.98.1:41049)
I20260925 04:38:57.047758 2299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "626b1f46eca84651875ead9fd8e747f1" candidate_uuid: "90534941bd5549558e508bdb29ac8265" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc188c63638c472e899fcfa41e747aad"
I20260925 04:38:57.047758 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "626b1f46eca84651875ead9fd8e747f1" candidate_uuid: "90534941bd5549558e508bdb29ac8265" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c1bfb87491488db8b4877520e5ac60"
I20260925 04:38:57.047924 2579 raft_consensus.cc:3037] T 626b1f46eca84651875ead9fd8e747f1 P 41c1bfb87491488db8b4877520e5ac60 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:38:57.047920 2299 raft_consensus.cc:3037] T 626b1f46eca84651875ead9fd8e747f1 P fc188c63638c472e899fcfa41e747aad [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:38:57.049288 2299 raft_consensus.cc:2454] T 626b1f46eca84651875ead9fd8e747f1 P fc188c63638c472e899fcfa41e747aad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90534941bd5549558e508bdb29ac8265 in term 1.
I20260925 04:38:57.049350 2579 raft_consensus.cc:2454] T 626b1f46eca84651875ead9fd8e747f1 P 41c1bfb87491488db8b4877520e5ac60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90534941bd5549558e508bdb29ac8265 in term 1.
I20260925 04:38:57.049659 2372 leader_election.cc:302] T 626b1f46eca84651875ead9fd8e747f1 P 90534941bd5549558e508bdb29ac8265 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90534941bd5549558e508bdb29ac8265, fc188c63638c472e899fcfa41e747aad; no voters:
I20260925 04:38:57.049825 2656 raft_consensus.cc:2781] T 626b1f46eca84651875ead9fd8e747f1 P 90534941bd5549558e508bdb29ac8265 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:38:57.050151 2656 raft_consensus.cc:686] T 626b1f46eca84651875ead9fd8e747f1 P 90534941bd5549558e508bdb29ac8265 [term 1 LEADER]: Becoming Leader. State: Replica: 90534941bd5549558e508bdb29ac8265, State: Running, Role: LEADER
I20260925 04:38:57.050535 2656 consensus_queue.cc:237] T 626b1f46eca84651875ead9fd8e747f1 P 90534941bd5549558e508bdb29ac8265 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90534941bd5549558e508bdb29ac8265" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "41c1bfb87491488db8b4877520e5ac60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 } } peers { permanent_uuid: "fc188c63638c472e899fcfa41e747aad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41049 } }
I20260925 04:38:57.054288 2141 catalog_manager.cc:5495] T 626b1f46eca84651875ead9fd8e747f1 P 90534941bd5549558e508bdb29ac8265 reported cstate change: term changed from 0 to 1, leader changed from <none> to 90534941bd5549558e508bdb29ac8265 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "90534941bd5549558e508bdb29ac8265" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90534941bd5549558e508bdb29ac8265" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "41c1bfb87491488db8b4877520e5ac60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc188c63638c472e899fcfa41e747aad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41049 } health_report { overall_health: UNKNOWN } } }
W20260925 04:38:57.063930 2626 tablet.cc:2369] T 626b1f46eca84651875ead9fd8e747f1 P 41c1bfb87491488db8b4877520e5ac60: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 04:38:57.083906 2486 tablet.cc:2369] T 626b1f46eca84651875ead9fd8e747f1 P 90534941bd5549558e508bdb29ac8265: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 04:38:57.100003 2346 tablet.cc:2369] T 626b1f46eca84651875ead9fd8e747f1 P fc188c63638c472e899fcfa41e747aad: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 04:38:57 dist-test-slave-dc95 krb5kdc[2097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311135, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:38:57.137701 2665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:57.129422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38605 (local address 127.0.0.1:37988)
0925 04:38:57.129747 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:57.129759 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:38:57.129908 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:38:57.130128 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:38:57.130131 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:38:57.130620 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:38:57.130728 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:57.130736 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:57.131482 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:57.131486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:57.132063 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:57.132071 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:57.132471 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:57.132475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:57.132548 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:57.133118 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:38:57.133131 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:38:57.134611 (+ 1480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:38:57.136269 (+ 1658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:57.136278 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:57.136283 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:57.136541 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:57.136788 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:57.136790 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:57.136792 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:57.136864 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:57.137175 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:38:57.137180 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:38:57.137290 (+ 110us) client_negotiation.cc:769] Sending connection context
0925 04:38:57.137380 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 04:38:57.137538 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":121,"threads_started":1}
I20260925 04:38:57.137722 2191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:57.129427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37988 (local address 127.0.98.62:38605)
0925 04:38:57.129588 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:57.129593 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:57.129839 (+ 246us) server_negotiation.cc:407] Connection header received
0925 04:38:57.129994 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:57.129998 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:57.130033 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:57.130099 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:57.130860 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:57.131383 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:57.132170 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:57.132373 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:57.132456 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:57.134765 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:38:57.134768 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:38:57.134770 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:38:57.134793 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:38:57.136176 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:57.136636 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:57.136639 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:57.136640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:57.136714 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:57.136922 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:57.136924 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:57.136925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:57.137094 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:38:57.137164 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:57.137427 (+ 263us) server_negotiation.cc:299] Negotiation successful
0925 04:38:57.137598 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":73}
I20260925 04:38:57.141588 2141 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37988:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 04:38:57.142340 2141 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:38:57.147312 2279 tablet_service.cc:1467] Processing CreateTablet for tablet 8c193ec457bf4439b4444a3ebcc79ef6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f8758422ca1e4427ae43412613ad1a11]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 04:38:57.147354 2559 tablet_service.cc:1467] Processing CreateTablet for tablet 8c193ec457bf4439b4444a3ebcc79ef6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f8758422ca1e4427ae43412613ad1a11]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 04:38:57.147294 2419 tablet_service.cc:1467] Processing CreateTablet for tablet 8c193ec457bf4439b4444a3ebcc79ef6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f8758422ca1e4427ae43412613ad1a11]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 04:38:57.148151 2279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c193ec457bf4439b4444a3ebcc79ef6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:38:57.148151 2559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c193ec457bf4439b4444a3ebcc79ef6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:38:57.148195 2419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c193ec457bf4439b4444a3ebcc79ef6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:38:57.151062 2648 tablet_bootstrap.cc:492] T 8c193ec457bf4439b4444a3ebcc79ef6 P fc188c63638c472e899fcfa41e747aad: Bootstrap starting.
I20260925 04:38:57.151800 2649 tablet_bootstrap.cc:492] T 8c193ec457bf4439b4444a3ebcc79ef6 P 41c1bfb87491488db8b4877520e5ac60: Bootstrap starting.
I20260925 04:38:57.152091 2648 tablet_bootstrap.cc:654] T 8c193ec457bf4439b4444a3ebcc79ef6 P fc188c63638c472e899fcfa41e747aad: Neither blocks nor log segments found. Creating new log.
I20260925 04:38:57.152678 2649 tablet_bootstrap.cc:654] T 8c193ec457bf4439b4444a3ebcc79ef6 P 41c1bfb87491488db8b4877520e5ac60: Neither blocks nor log segments found. Creating new log.
I20260925 04:38:57.153337 2650 tablet_bootstrap.cc:492] T 8c193ec457bf4439b4444a3ebcc79ef6 P 90534941bd5549558e508bdb29ac8265: Bootstrap starting.
I20260925 04:38:57.153347 2648 tablet_bootstrap.cc:492] T 8c193ec457bf4439b4444a3ebcc79ef6 P fc188c63638c472e899fcfa41e747aad: No bootstrap required, opened a new log
I20260925 04:38:57.153442 2648 ts_tablet_manager.cc:1397] T 8c193ec457bf4439b4444a3ebcc79ef6 P fc188c63638c472e899fcfa41e747aad: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 04:38:57.153877 2648 raft_consensus.cc:348] T 8c193ec457bf4439b4444a3ebcc79ef6 P fc188c63638c472e899fcfa41e747aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc188c63638c472e899fcfa41e747aad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41049 } } peers { permanent_uuid: "90534941bd5549558e508bdb29ac8265" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "41c1bfb87491488db8b4877520e5ac60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 } }
I20260925 04:38:57.153980 2648 raft_consensus.cc:374] T 8c193ec457bf4439b4444a3ebcc79ef6 P fc188c63638c472e899fcfa41e747aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:38:57.154000 2648 raft_consensus.cc:729] T 8c193ec457bf4439b4444a3ebcc79ef6 P fc188c63638c472e899fcfa41e747aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc188c63638c472e899fcfa41e747aad, State: Initialized, Role: FOLLOWER
I20260925 04:38:57.154095 2649 tablet_bootstrap.cc:492] T 8c193ec457bf4439b4444a3ebcc79ef6 P 41c1bfb87491488db8b4877520e5ac60: No bootstrap required, opened a new log
I20260925 04:38:57.154106 2648 consensus_queue.cc:260] T 8c193ec457bf4439b4444a3ebcc79ef6 P fc188c63638c472e899fcfa41e747aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc188c63638c472e899fcfa41e747aad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41049 } } peers { permanent_uuid: "90534941bd5549558e508bdb29ac8265" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "41c1bfb87491488db8b4877520e5ac60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 } }
I20260925 04:38:57.154172 2649 ts_tablet_manager.cc:1397] T 8c193ec457bf4439b4444a3ebcc79ef6 P 41c1bfb87491488db8b4877520e5ac60: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 04:38:57.154224 2650 tablet_bootstrap.cc:654] T 8c193ec457bf4439b4444a3ebcc79ef6 P 90534941bd5549558e508bdb29ac8265: Neither blocks nor log segments found. Creating new log.
I20260925 04:38:57.154343 2648 ts_tablet_manager.cc:1428] T 8c193ec457bf4439b4444a3ebcc79ef6 P fc188c63638c472e899fcfa41e747aad: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260925 04:38:57.154350 2655 tablet_replica.cc:439] T 8c193ec457bf4439b4444a3ebcc79ef6 P fc188c63638c472e899fcfa41e747aad: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc188c63638c472e899fcfa41e747aad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41049 } } peers { permanent_uuid: "90534941bd5549558e508bdb29ac8265" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "41c1bfb87491488db8b4877520e5ac60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 } } }
I20260925 04:38:57.154515 2655 tablet_replica.cc:442] T 8c193ec457bf4439b4444a3ebcc79ef6 P fc188c63638c472e899fcfa41e747aad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:38:57.154623 2649 raft_consensus.cc:348] T 8c193ec457bf4439b4444a3ebcc79ef6 P 41c1bfb87491488db8b4877520e5ac60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc188c63638c472e899fcfa41e747aad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41049 } } peers { permanent_uuid: "90534941bd5549558e508bdb29ac8265" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "41c1bfb87491488db8b4877520e5ac60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 } }
I20260925 04:38:57.154726 2649 raft_consensus.cc:374] T 8c193ec457bf4439b4444a3ebcc79ef6 P 41c1bfb87491488db8b4877520e5ac60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:38:57.154757 2649 raft_consensus.cc:729] T 8c193ec457bf4439b4444a3ebcc79ef6 P 41c1bfb87491488db8b4877520e5ac60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c1bfb87491488db8b4877520e5ac60, State: Initialized, Role: FOLLOWER
I20260925 04:38:57.154871 2649 consensus_queue.cc:260] T 8c193ec457bf4439b4444a3ebcc79ef6 P 41c1bfb87491488db8b4877520e5ac60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc188c63638c472e899fcfa41e747aad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41049 } } peers { permanent_uuid: "90534941bd5549558e508bdb29ac8265" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "41c1bfb87491488db8b4877520e5ac60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 } }
I20260925 04:38:57.155098 2649 ts_tablet_manager.cc:1428] T 8c193ec457bf4439b4444a3ebcc79ef6 P 41c1bfb87491488db8b4877520e5ac60: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 04:38:57.155332 2654 tablet_replica.cc:439] T 8c193ec457bf4439b4444a3ebcc79ef6 P 41c1bfb87491488db8b4877520e5ac60: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc188c63638c472e899fcfa41e747aad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41049 } } peers { permanent_uuid: "90534941bd5549558e508bdb29ac8265" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "41c1bfb87491488db8b4877520e5ac60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 } } }
I20260925 04:38:57.155493 2654 tablet_replica.cc:442] T 8c193ec457bf4439b4444a3ebcc79ef6 P 41c1bfb87491488db8b4877520e5ac60: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:38:57.155848 2650 tablet_bootstrap.cc:492] T 8c193ec457bf4439b4444a3ebcc79ef6 P 90534941bd5549558e508bdb29ac8265: No bootstrap required, opened a new log
I20260925 04:38:57.155942 2650 ts_tablet_manager.cc:1397] T 8c193ec457bf4439b4444a3ebcc79ef6 P 90534941bd5549558e508bdb29ac8265: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 04:38:57.156394 2650 raft_consensus.cc:348] T 8c193ec457bf4439b4444a3ebcc79ef6 P 90534941bd5549558e508bdb29ac8265 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc188c63638c472e899fcfa41e747aad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41049 } } peers { permanent_uuid: "90534941bd5549558e508bdb29ac8265" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "41c1bfb87491488db8b4877520e5ac60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 } }
I20260925 04:38:57.156507 2650 raft_consensus.cc:374] T 8c193ec457bf4439b4444a3ebcc79ef6 P 90534941bd5549558e508bdb29ac8265 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:38:57.156539 2650 raft_consensus.cc:729] T 8c193ec457bf4439b4444a3ebcc79ef6 P 90534941bd5549558e508bdb29ac8265 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90534941bd5549558e508bdb29ac8265, State: Initialized, Role: FOLLOWER
I20260925 04:38:57.156659 2650 consensus_queue.cc:260] T 8c193ec457bf4439b4444a3ebcc79ef6 P 90534941bd5549558e508bdb29ac8265 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc188c63638c472e899fcfa41e747aad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41049 } } peers { permanent_uuid: "90534941bd5549558e508bdb29ac8265" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "41c1bfb87491488db8b4877520e5ac60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 } }
I20260925 04:38:57.156941 2650 ts_tablet_manager.cc:1428] T 8c193ec457bf4439b4444a3ebcc79ef6 P 90534941bd5549558e508bdb29ac8265: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 04:38:57.156919 2656 tablet_replica.cc:439] T 8c193ec457bf4439b4444a3ebcc79ef6 P 90534941bd5549558e508bdb29ac8265: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc188c63638c472e899fcfa41e747aad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41049 } } peers { permanent_uuid: "90534941bd5549558e508bdb29ac8265" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "41c1bfb87491488db8b4877520e5ac60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 } } }
I20260925 04:38:57.157053 2656 tablet_replica.cc:442] T 8c193ec457bf4439b4444a3ebcc79ef6 P 90534941bd5549558e508bdb29ac8265: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:38:57.418349 2654 raft_consensus.cc:482] T 8c193ec457bf4439b4444a3ebcc79ef6 P 41c1bfb87491488db8b4877520e5ac60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:38:57.418514 2654 raft_consensus.cc:504] T 8c193ec457bf4439b4444a3ebcc79ef6 P 41c1bfb87491488db8b4877520e5ac60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc188c63638c472e899fcfa41e747aad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41049 } } peers { permanent_uuid: "90534941bd5549558e508bdb29ac8265" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "41c1bfb87491488db8b4877520e5ac60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 } }
I20260925 04:38:57.419771 2654 leader_election.cc:288] T 8c193ec457bf4439b4444a3ebcc79ef6 P 41c1bfb87491488db8b4877520e5ac60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc188c63638c472e899fcfa41e747aad (127.0.98.1:41049), 90534941bd5549558e508bdb29ac8265 (127.0.98.2:41491)
I20260925 04:38:57.423682 2671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:57.420177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51367 (local address 127.0.98.2:41491)
0925 04:38:57.420461 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:57.420477 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:57.420493 (+ 16us) server_negotiation.cc:407] Connection header received
0925 04:38:57.420641 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:57.420644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:57.420685 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:57.420755 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:38:57.421058 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:57.421564 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:57.422641 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:57.423059 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:57.423156 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:57.423224 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:57.423408 (+ 184us) server_negotiation.cc:299] Negotiation successful
0925 04:38:57.423484 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":56,"threads_started":1}
I20260925 04:38:57.423707 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:57.420277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42589 (local address 127.0.98.1:41049)
0925 04:38:57.420393 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:57.420397 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:57.420476 (+ 79us) server_negotiation.cc:407] Connection header received
0925 04:38:57.420637 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:57.420641 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:57.420690 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:57.420750 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:38:57.421027 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:57.421520 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:57.422683 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:57.423093 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:57.423184 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:57.423245 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:57.423464 (+ 219us) server_negotiation.cc:299] Negotiation successful
0925 04:38:57.423545 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 04:38:57.423719 2670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:57.420055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41491 (local address 127.0.98.3:51367)
0925 04:38:57.420381 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:57.420396 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:38:57.420536 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:38:57.420774 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:38:57.420777 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:38:57.420800 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:38:57.420948 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:57.420953 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:57.421665 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:57.421668 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:57.422511 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:57.422522 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:57.423177 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:57.423181 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:57.423281 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:57.423305 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 04:38:57.423372 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 04:38:57.423507 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":4,"spinlock_wait_cycles":36736,"thread_start_us":77,"threads_started":1}
I20260925 04:38:57.423732 2669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:57.420009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41049 (local address 127.0.98.3:42589)
0925 04:38:57.420376 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:57.420392 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:38:57.420537 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:38:57.420776 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:38:57.420778 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:38:57.420802 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:38:57.420931 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:57.420936 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:57.421626 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:57.421630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:57.422560 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:57.422570 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:57.423227 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:57.423230 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:57.423330 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:57.423355 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 04:38:57.423422 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 04:38:57.423510 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"mutex_wait_us":63,"spinlock_wait_cycles":8320,"thread_start_us":88,"threads_started":1}
I20260925 04:38:57.424173 2299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c193ec457bf4439b4444a3ebcc79ef6" candidate_uuid: "41c1bfb87491488db8b4877520e5ac60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc188c63638c472e899fcfa41e747aad" is_pre_election: true
I20260925 04:38:57.424319 2299 raft_consensus.cc:2454] T 8c193ec457bf4439b4444a3ebcc79ef6 P fc188c63638c472e899fcfa41e747aad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41c1bfb87491488db8b4877520e5ac60 in term 0.
I20260925 04:38:57.424611 2435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c193ec457bf4439b4444a3ebcc79ef6" candidate_uuid: "41c1bfb87491488db8b4877520e5ac60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90534941bd5549558e508bdb29ac8265" is_pre_election: true
I20260925 04:38:57.424682 2512 leader_election.cc:302] T 8c193ec457bf4439b4444a3ebcc79ef6 P 41c1bfb87491488db8b4877520e5ac60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41c1bfb87491488db8b4877520e5ac60, fc188c63638c472e899fcfa41e747aad; no voters:
I20260925 04:38:57.424823 2435 raft_consensus.cc:2454] T 8c193ec457bf4439b4444a3ebcc79ef6 P 90534941bd5549558e508bdb29ac8265 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41c1bfb87491488db8b4877520e5ac60 in term 0.
I20260925 04:38:57.425045 2654 raft_consensus.cc:2781] T 8c193ec457bf4439b4444a3ebcc79ef6 P 41c1bfb87491488db8b4877520e5ac60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:38:57.425113 2654 raft_consensus.cc:482] T 8c193ec457bf4439b4444a3ebcc79ef6 P 41c1bfb87491488db8b4877520e5ac60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:38:57.425158 2654 raft_consensus.cc:3037] T 8c193ec457bf4439b4444a3ebcc79ef6 P 41c1bfb87491488db8b4877520e5ac60 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:38:57.426136 2654 raft_consensus.cc:504] T 8c193ec457bf4439b4444a3ebcc79ef6 P 41c1bfb87491488db8b4877520e5ac60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc188c63638c472e899fcfa41e747aad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41049 } } peers { permanent_uuid: "90534941bd5549558e508bdb29ac8265" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "41c1bfb87491488db8b4877520e5ac60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 } }
I20260925 04:38:57.426481 2654 leader_election.cc:288] T 8c193ec457bf4439b4444a3ebcc79ef6 P 41c1bfb87491488db8b4877520e5ac60 [CANDIDATE]: Term 1 election: Requested vote from peers fc188c63638c472e899fcfa41e747aad (127.0.98.1:41049), 90534941bd5549558e508bdb29ac8265 (127.0.98.2:41491)
I20260925 04:38:57.426970 2299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c193ec457bf4439b4444a3ebcc79ef6" candidate_uuid: "41c1bfb87491488db8b4877520e5ac60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc188c63638c472e899fcfa41e747aad"
I20260925 04:38:57.427134 2299 raft_consensus.cc:3037] T 8c193ec457bf4439b4444a3ebcc79ef6 P fc188c63638c472e899fcfa41e747aad [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:38:57.427752 2435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c193ec457bf4439b4444a3ebcc79ef6" candidate_uuid: "41c1bfb87491488db8b4877520e5ac60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90534941bd5549558e508bdb29ac8265"
I20260925 04:38:57.427873 2435 raft_consensus.cc:3037] T 8c193ec457bf4439b4444a3ebcc79ef6 P 90534941bd5549558e508bdb29ac8265 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:38:57.428136 2299 raft_consensus.cc:2454] T 8c193ec457bf4439b4444a3ebcc79ef6 P fc188c63638c472e899fcfa41e747aad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41c1bfb87491488db8b4877520e5ac60 in term 1.
I20260925 04:38:57.428440 2512 leader_election.cc:302] T 8c193ec457bf4439b4444a3ebcc79ef6 P 41c1bfb87491488db8b4877520e5ac60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41c1bfb87491488db8b4877520e5ac60, fc188c63638c472e899fcfa41e747aad; no voters:
I20260925 04:38:57.428614 2654 raft_consensus.cc:2781] T 8c193ec457bf4439b4444a3ebcc79ef6 P 41c1bfb87491488db8b4877520e5ac60 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:38:57.428776 2654 raft_consensus.cc:686] T 8c193ec457bf4439b4444a3ebcc79ef6 P 41c1bfb87491488db8b4877520e5ac60 [term 1 LEADER]: Becoming Leader. State: Replica: 41c1bfb87491488db8b4877520e5ac60, State: Running, Role: LEADER
I20260925 04:38:57.428879 2435 raft_consensus.cc:2454] T 8c193ec457bf4439b4444a3ebcc79ef6 P 90534941bd5549558e508bdb29ac8265 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41c1bfb87491488db8b4877520e5ac60 in term 1.
I20260925 04:38:57.429307 2654 consensus_queue.cc:237] T 8c193ec457bf4439b4444a3ebcc79ef6 P 41c1bfb87491488db8b4877520e5ac60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc188c63638c472e899fcfa41e747aad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41049 } } peers { permanent_uuid: "90534941bd5549558e508bdb29ac8265" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "41c1bfb87491488db8b4877520e5ac60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 } }
I20260925 04:38:57.431281 2672 tablet_replica.cc:439] T 8c193ec457bf4439b4444a3ebcc79ef6 P 41c1bfb87491488db8b4877520e5ac60: TxnStatusTablet state changed. Reason: New leader 41c1bfb87491488db8b4877520e5ac60. Latest consensus state: current_term: 1 leader_uuid: "41c1bfb87491488db8b4877520e5ac60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc188c63638c472e899fcfa41e747aad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41049 } } peers { permanent_uuid: "90534941bd5549558e508bdb29ac8265" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "41c1bfb87491488db8b4877520e5ac60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 } } }
I20260925 04:38:57.431425 2672 tablet_replica.cc:442] T 8c193ec457bf4439b4444a3ebcc79ef6 P 41c1bfb87491488db8b4877520e5ac60: This TxnStatusTablet replica's current role is: LEADER
I20260925 04:38:57.431947 2674 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 04:38:57.432143 2674 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 04:38:57.432668 2141 catalog_manager.cc:5495] T 8c193ec457bf4439b4444a3ebcc79ef6 P 41c1bfb87491488db8b4877520e5ac60 reported cstate change: term changed from 0 to 1, leader changed from <none> to 41c1bfb87491488db8b4877520e5ac60 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "41c1bfb87491488db8b4877520e5ac60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc188c63638c472e899fcfa41e747aad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90534941bd5549558e508bdb29ac8265" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41c1bfb87491488db8b4877520e5ac60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 } health_report { overall_health: HEALTHY } } }
I20260925 04:38:57.451208 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:57.447087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36108 (local address 127.0.98.3:38467)
0925 04:38:57.447215 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:57.447219 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:57.447308 (+ 89us) server_negotiation.cc:407] Connection header received
0925 04:38:57.447499 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:57.447501 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:57.447560 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:57.447707 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:38:57.447931 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:57.448470 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:57.449722 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:57.449946 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:57.450036 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:57.450356 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:38:57.450857 (+ 501us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:38:57.450951 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:57.451036 (+ 85us) server_negotiation.cc:299] Negotiation successful
0925 04:38:57.451092 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 04:38:57.451285 2665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:57.446975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38467 (local address 127.0.0.1:36108)
0925 04:38:57.447238 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:57.447253 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:38:57.447372 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:38:57.447727 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:38:57.447730 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:38:57.447738 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 04:38:57.447849 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:57.447855 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:57.448569 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:57.448572 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:57.449580 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:57.449592 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:57.450057 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:57.450061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:57.450152 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:57.450213 (+ 61us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 04:38:57.450949 (+ 736us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 04:38:57.450963 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 04:38:57.451030 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 04:38:57.451137 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156}
I20260925 04:38:57.457304 2299 raft_consensus.cc:1261] T 8c193ec457bf4439b4444a3ebcc79ef6 P fc188c63638c472e899fcfa41e747aad [term 1 FOLLOWER]: Refusing update from remote peer 41c1bfb87491488db8b4877520e5ac60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:38:57.457301 2435 raft_consensus.cc:1261] T 8c193ec457bf4439b4444a3ebcc79ef6 P 90534941bd5549558e508bdb29ac8265 [term 1 FOLLOWER]: Refusing update from remote peer 41c1bfb87491488db8b4877520e5ac60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:38:57.457991 2654 consensus_queue.cc:1035] T 8c193ec457bf4439b4444a3ebcc79ef6 P 41c1bfb87491488db8b4877520e5ac60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc188c63638c472e899fcfa41e747aad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:38:57.458216 2672 consensus_queue.cc:1035] T 8c193ec457bf4439b4444a3ebcc79ef6 P 41c1bfb87491488db8b4877520e5ac60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90534941bd5549558e508bdb29ac8265" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:38:57.461730 2656 tablet_replica.cc:439] T 8c193ec457bf4439b4444a3ebcc79ef6 P 90534941bd5549558e508bdb29ac8265: TxnStatusTablet state changed. Reason: New leader 41c1bfb87491488db8b4877520e5ac60. Latest consensus state: current_term: 1 leader_uuid: "41c1bfb87491488db8b4877520e5ac60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc188c63638c472e899fcfa41e747aad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41049 } } peers { permanent_uuid: "90534941bd5549558e508bdb29ac8265" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "41c1bfb87491488db8b4877520e5ac60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 } } }
I20260925 04:38:57.461884 2656 tablet_replica.cc:442] T 8c193ec457bf4439b4444a3ebcc79ef6 P 90534941bd5549558e508bdb29ac8265: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:38:57.462225 2655 tablet_replica.cc:439] T 8c193ec457bf4439b4444a3ebcc79ef6 P fc188c63638c472e899fcfa41e747aad: TxnStatusTablet state changed. Reason: New leader 41c1bfb87491488db8b4877520e5ac60. Latest consensus state: current_term: 1 leader_uuid: "41c1bfb87491488db8b4877520e5ac60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc188c63638c472e899fcfa41e747aad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41049 } } peers { permanent_uuid: "90534941bd5549558e508bdb29ac8265" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "41c1bfb87491488db8b4877520e5ac60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 } } }
I20260925 04:38:57.462339 2655 tablet_replica.cc:442] T 8c193ec457bf4439b4444a3ebcc79ef6 P fc188c63638c472e899fcfa41e747aad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:38:57.466784 2656 tablet_replica.cc:439] T 8c193ec457bf4439b4444a3ebcc79ef6 P 90534941bd5549558e508bdb29ac8265: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "41c1bfb87491488db8b4877520e5ac60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc188c63638c472e899fcfa41e747aad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41049 } } peers { permanent_uuid: "90534941bd5549558e508bdb29ac8265" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "41c1bfb87491488db8b4877520e5ac60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 } } }
I20260925 04:38:57.466930 2656 tablet_replica.cc:442] T 8c193ec457bf4439b4444a3ebcc79ef6 P 90534941bd5549558e508bdb29ac8265: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:38:57.466900 2655 tablet_replica.cc:439] T 8c193ec457bf4439b4444a3ebcc79ef6 P fc188c63638c472e899fcfa41e747aad: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "41c1bfb87491488db8b4877520e5ac60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc188c63638c472e899fcfa41e747aad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41049 } } peers { permanent_uuid: "90534941bd5549558e508bdb29ac8265" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "41c1bfb87491488db8b4877520e5ac60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 } } }
I20260925 04:38:57.467025 2655 tablet_replica.cc:442] T 8c193ec457bf4439b4444a3ebcc79ef6 P fc188c63638c472e899fcfa41e747aad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:38:57.467724 2678 mvcc.cc:205] Tried to move back new op lower bound from 7333114419017019392 to 7333114418912927744. Current Snapshot: MvccSnapshot[applied={T|T < 7333114419017019392}]
I20260925 04:38:57.468343 2682 tablet_replica.cc:439] T 8c193ec457bf4439b4444a3ebcc79ef6 P 41c1bfb87491488db8b4877520e5ac60: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "41c1bfb87491488db8b4877520e5ac60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc188c63638c472e899fcfa41e747aad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41049 } } peers { permanent_uuid: "90534941bd5549558e508bdb29ac8265" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "41c1bfb87491488db8b4877520e5ac60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 } } }
I20260925 04:38:57.468598 2672 tablet_replica.cc:439] T 8c193ec457bf4439b4444a3ebcc79ef6 P 41c1bfb87491488db8b4877520e5ac60: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "41c1bfb87491488db8b4877520e5ac60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc188c63638c472e899fcfa41e747aad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41049 } } peers { permanent_uuid: "90534941bd5549558e508bdb29ac8265" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "41c1bfb87491488db8b4877520e5ac60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 } } }
I20260925 04:38:57.468720 2672 tablet_replica.cc:442] T 8c193ec457bf4439b4444a3ebcc79ef6 P 41c1bfb87491488db8b4877520e5ac60: This TxnStatusTablet replica's current role is: LEADER
I20260925 04:38:57.468600 2682 tablet_replica.cc:442] T 8c193ec457bf4439b4444a3ebcc79ef6 P 41c1bfb87491488db8b4877520e5ac60: This TxnStatusTablet replica's current role is: LEADER
I20260925 04:38:57.482810 2671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:57.478008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41964 (local address 127.0.98.2:41491)
0925 04:38:57.478147 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:57.478151 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:57.478259 (+ 108us) server_negotiation.cc:407] Connection header received
0925 04:38:57.478409 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:57.478412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:57.478491 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:57.478624 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:38:57.478935 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:57.479969 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:57.480946 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:57.481288 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:57.481393 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:57.481692 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:38:57.482320 (+ 628us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:38:57.482441 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:57.482584 (+ 143us) server_negotiation.cc:299] Negotiation successful
0925 04:38:57.482657 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 04:38:57.495146 2656 consensus_queue.cc:1035] T 626b1f46eca84651875ead9fd8e747f1 P 90534941bd5549558e508bdb29ac8265 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc188c63638c472e899fcfa41e747aad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:38:57.495342 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:57.491390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39367 (local address 127.0.98.3:38467)
0925 04:38:57.491531 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:57.491536 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:57.491732 (+ 196us) server_negotiation.cc:407] Connection header received
0925 04:38:57.491896 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:57.491898 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:57.491946 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:57.492008 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:38:57.492268 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:57.492764 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:57.494050 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:57.494375 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:57.494482 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:57.494883 (+ 401us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:38:57.494979 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:38:57.495053 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:57.495150 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 04:38:57.495210 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 04:38:57.495388 2691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:57.491286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38467 (local address 127.0.98.2:39367)
0925 04:38:57.491656 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:57.491672 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:38:57.491801 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:38:57.492031 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:38:57.492034 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:38:57.492041 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 04:38:57.492191 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:57.492197 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:57.492854 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:57.492858 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:57.493916 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:57.493930 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:57.494505 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:57.494509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:57.494619 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:57.494676 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 04:38:57.495064 (+ 388us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 04:38:57.495070 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 04:38:57.495127 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 04:38:57.495215 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":121,"threads_started":1}
I20260925 04:38:57.500223 2658 consensus_queue.cc:1035] T 626b1f46eca84651875ead9fd8e747f1 P 90534941bd5549558e508bdb29ac8265 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c1bfb87491488db8b4877520e5ac60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:38:57.528155 2671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:57.524035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38595 (local address 127.0.98.2:41491)
0925 04:38:57.524156 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:57.524160 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:57.524463 (+ 303us) server_negotiation.cc:407] Connection header received
0925 04:38:57.524735 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:57.524739 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:57.524796 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:57.524877 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:38:57.525148 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:57.525675 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:57.526939 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:57.527160 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:57.527242 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:57.527670 (+ 428us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:38:57.527760 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:38:57.527825 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:57.527930 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 04:38:57.527989 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 04:38:57.528167 2695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:57.523898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41491 (local address 127.0.98.3:38595)
0925 04:38:57.524383 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:57.524398 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:38:57.524610 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:38:57.524898 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:38:57.524901 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:38:57.524908 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 04:38:57.525038 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:57.525044 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:57.525775 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:57.525778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:57.526794 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:57.526807 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:57.527266 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:57.527279 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:57.527383 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:57.527443 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 04:38:57.527846 (+ 403us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 04:38:57.527852 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 04:38:57.527922 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 04:38:57.528008 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":126,"threads_started":1}
I20260925 04:38:57.556252 2140 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37978:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:38:57.556469 2140 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37978:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:38:57.559588 2140 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3d4badfd4e7f4245a022ad52c0385cce: Sending DeleteTablet for 3 replicas of tablet 626b1f46eca84651875ead9fd8e747f1
I20260925 04:38:57.560616 2559 tablet_service.cc:1514] Processing DeleteTablet for tablet 626b1f46eca84651875ead9fd8e747f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:38:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36102
I20260925 04:38:57.561350 2419 tablet_service.cc:1514] Processing DeleteTablet for tablet 626b1f46eca84651875ead9fd8e747f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:38:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41962
I20260925 04:38:57.561403 2140 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37978:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:38:57.561878 2140 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:38:57.562225 2279 tablet_service.cc:1514] Processing DeleteTablet for tablet 626b1f46eca84651875ead9fd8e747f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:38:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37366
I20260925 04:38:57.562812 2701 tablet_replica.cc:330] T 626b1f46eca84651875ead9fd8e747f1 P fc188c63638c472e899fcfa41e747aad: stopping tablet replica
I20260925 04:38:57.563177 2701 raft_consensus.cc:2229] T 626b1f46eca84651875ead9fd8e747f1 P fc188c63638c472e899fcfa41e747aad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:38:57.563443 2701 raft_consensus.cc:2258] T 626b1f46eca84651875ead9fd8e747f1 P fc188c63638c472e899fcfa41e747aad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:38:57.564258 2700 tablet_replica.cc:330] T 626b1f46eca84651875ead9fd8e747f1 P 90534941bd5549558e508bdb29ac8265: stopping tablet replica
I20260925 04:38:57.564667 2700 raft_consensus.cc:2229] T 626b1f46eca84651875ead9fd8e747f1 P 90534941bd5549558e508bdb29ac8265 [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:38:57.565073 2699 tablet_replica.cc:330] T 626b1f46eca84651875ead9fd8e747f1 P 41c1bfb87491488db8b4877520e5ac60: stopping tablet replica
I20260925 04:38:57.565605 2700 raft_consensus.cc:2258] T 626b1f46eca84651875ead9fd8e747f1 P 90534941bd5549558e508bdb29ac8265 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:38:57.566250 2701 ts_tablet_manager.cc:1905] T 626b1f46eca84651875ead9fd8e747f1 P fc188c63638c472e899fcfa41e747aad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:38:57.567895 2279 tablet_service.cc:1467] Processing CreateTablet for tablet 5aaf797144b043008f0f58c79b4e081c (DEFAULT_TABLE table=test-table [id=53673001fc084072ac41d8d52420d59d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:38:57.568174 2279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5aaf797144b043008f0f58c79b4e081c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:38:57.568156 2700 ts_tablet_manager.cc:1905] T 626b1f46eca84651875ead9fd8e747f1 P 90534941bd5549558e508bdb29ac8265: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:38:57.570461 2701 ts_tablet_manager.cc:1918] T 626b1f46eca84651875ead9fd8e747f1 P fc188c63638c472e899fcfa41e747aad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 04:38:57.570551 2701 log.cc:1196] T 626b1f46eca84651875ead9fd8e747f1 P fc188c63638c472e899fcfa41e747aad: Deleting WAL directory at /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/ts-0/wal/wals/626b1f46eca84651875ead9fd8e747f1
I20260925 04:38:57.570673 2419 tablet_service.cc:1467] Processing CreateTablet for tablet 5aaf797144b043008f0f58c79b4e081c (DEFAULT_TABLE table=test-table [id=53673001fc084072ac41d8d52420d59d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:38:57.570832 2701 ts_tablet_manager.cc:1939] T 626b1f46eca84651875ead9fd8e747f1 P fc188c63638c472e899fcfa41e747aad: Deleting consensus metadata
I20260925 04:38:57.571004 2419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5aaf797144b043008f0f58c79b4e081c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:38:57.572280 2700 ts_tablet_manager.cc:1918] T 626b1f46eca84651875ead9fd8e747f1 P 90534941bd5549558e508bdb29ac8265: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 04:38:57.572392 2700 log.cc:1196] T 626b1f46eca84651875ead9fd8e747f1 P 90534941bd5549558e508bdb29ac8265: Deleting WAL directory at /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/ts-1/wal/wals/626b1f46eca84651875ead9fd8e747f1
I20260925 04:38:57.573503 2648 tablet_bootstrap.cc:492] T 5aaf797144b043008f0f58c79b4e081c P fc188c63638c472e899fcfa41e747aad: Bootstrap starting.
I20260925 04:38:57.573654 2700 ts_tablet_manager.cc:1939] T 626b1f46eca84651875ead9fd8e747f1 P 90534941bd5549558e508bdb29ac8265: Deleting consensus metadata
I20260925 04:38:57.574540 2648 tablet_bootstrap.cc:654] T 5aaf797144b043008f0f58c79b4e081c P fc188c63638c472e899fcfa41e747aad: Neither blocks nor log segments found. Creating new log.
I20260925 04:38:57.575405 2650 tablet_bootstrap.cc:492] T 5aaf797144b043008f0f58c79b4e081c P 90534941bd5549558e508bdb29ac8265: Bootstrap starting.
I20260925 04:38:57.575778 2648 tablet_bootstrap.cc:492] T 5aaf797144b043008f0f58c79b4e081c P fc188c63638c472e899fcfa41e747aad: No bootstrap required, opened a new log
I20260925 04:38:57.575873 2648 ts_tablet_manager.cc:1397] T 5aaf797144b043008f0f58c79b4e081c P fc188c63638c472e899fcfa41e747aad: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260925 04:38:57.576320 2648 raft_consensus.cc:348] T 5aaf797144b043008f0f58c79b4e081c P fc188c63638c472e899fcfa41e747aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc188c63638c472e899fcfa41e747aad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41049 } } peers { permanent_uuid: "90534941bd5549558e508bdb29ac8265" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "41c1bfb87491488db8b4877520e5ac60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 } }
I20260925 04:38:57.576434 2648 raft_consensus.cc:374] T 5aaf797144b043008f0f58c79b4e081c P fc188c63638c472e899fcfa41e747aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:38:57.576431 2650 tablet_bootstrap.cc:654] T 5aaf797144b043008f0f58c79b4e081c P 90534941bd5549558e508bdb29ac8265: Neither blocks nor log segments found. Creating new log.
I20260925 04:38:57.576462 2648 raft_consensus.cc:729] T 5aaf797144b043008f0f58c79b4e081c P fc188c63638c472e899fcfa41e747aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc188c63638c472e899fcfa41e747aad, State: Initialized, Role: FOLLOWER
I20260925 04:38:57.576598 2648 consensus_queue.cc:260] T 5aaf797144b043008f0f58c79b4e081c P fc188c63638c472e899fcfa41e747aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc188c63638c472e899fcfa41e747aad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41049 } } peers { permanent_uuid: "90534941bd5549558e508bdb29ac8265" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "41c1bfb87491488db8b4877520e5ac60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 } }
I20260925 04:38:57.576864 2648 ts_tablet_manager.cc:1428] T 5aaf797144b043008f0f58c79b4e081c P fc188c63638c472e899fcfa41e747aad: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 04:38:57.578634 2126 catalog_manager.cc:4841] TS 90534941bd5549558e508bdb29ac8265 (127.0.98.2:41491): tablet 626b1f46eca84651875ead9fd8e747f1 (table test-table [id=c51d7b60229440a6a77d1a4d8beff158]) successfully deleted
I20260925 04:38:57.578882 2128 catalog_manager.cc:4841] TS fc188c63638c472e899fcfa41e747aad (127.0.98.1:41049): tablet 626b1f46eca84651875ead9fd8e747f1 (table test-table [id=c51d7b60229440a6a77d1a4d8beff158]) successfully deleted
I20260925 04:38:57.578953 2699 raft_consensus.cc:2229] T 626b1f46eca84651875ead9fd8e747f1 P 41c1bfb87491488db8b4877520e5ac60 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:38:57.579160 2699 raft_consensus.cc:2258] T 626b1f46eca84651875ead9fd8e747f1 P 41c1bfb87491488db8b4877520e5ac60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:38:57.579219 2559 tablet_service.cc:1467] Processing CreateTablet for tablet 5aaf797144b043008f0f58c79b4e081c (DEFAULT_TABLE table=test-table [id=53673001fc084072ac41d8d52420d59d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:38:57.579505 2559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5aaf797144b043008f0f58c79b4e081c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:38:57.580242 2650 tablet_bootstrap.cc:492] T 5aaf797144b043008f0f58c79b4e081c P 90534941bd5549558e508bdb29ac8265: No bootstrap required, opened a new log
I20260925 04:38:57.580348 2650 ts_tablet_manager.cc:1397] T 5aaf797144b043008f0f58c79b4e081c P 90534941bd5549558e508bdb29ac8265: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260925 04:38:57.580868 2650 raft_consensus.cc:348] T 5aaf797144b043008f0f58c79b4e081c P 90534941bd5549558e508bdb29ac8265 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc188c63638c472e899fcfa41e747aad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41049 } } peers { permanent_uuid: "90534941bd5549558e508bdb29ac8265" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "41c1bfb87491488db8b4877520e5ac60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 } }
I20260925 04:38:57.580986 2650 raft_consensus.cc:374] T 5aaf797144b043008f0f58c79b4e081c P 90534941bd5549558e508bdb29ac8265 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:38:57.581018 2650 raft_consensus.cc:729] T 5aaf797144b043008f0f58c79b4e081c P 90534941bd5549558e508bdb29ac8265 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90534941bd5549558e508bdb29ac8265, State: Initialized, Role: FOLLOWER
I20260925 04:38:57.581149 2650 consensus_queue.cc:260] T 5aaf797144b043008f0f58c79b4e081c P 90534941bd5549558e508bdb29ac8265 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc188c63638c472e899fcfa41e747aad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41049 } } peers { permanent_uuid: "90534941bd5549558e508bdb29ac8265" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "41c1bfb87491488db8b4877520e5ac60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 } }
I20260925 04:38:57.581456 2650 ts_tablet_manager.cc:1428] T 5aaf797144b043008f0f58c79b4e081c P 90534941bd5549558e508bdb29ac8265: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260925 04:38:57.582813 2649 tablet_bootstrap.cc:492] T 5aaf797144b043008f0f58c79b4e081c P 41c1bfb87491488db8b4877520e5ac60: Bootstrap starting.
I20260925 04:38:57.583849 2649 tablet_bootstrap.cc:654] T 5aaf797144b043008f0f58c79b4e081c P 41c1bfb87491488db8b4877520e5ac60: Neither blocks nor log segments found. Creating new log.
I20260925 04:38:57.585963 2649 tablet_bootstrap.cc:492] T 5aaf797144b043008f0f58c79b4e081c P 41c1bfb87491488db8b4877520e5ac60: No bootstrap required, opened a new log
I20260925 04:38:57.586059 2649 ts_tablet_manager.cc:1397] T 5aaf797144b043008f0f58c79b4e081c P 41c1bfb87491488db8b4877520e5ac60: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 04:38:57.586493 2649 raft_consensus.cc:348] T 5aaf797144b043008f0f58c79b4e081c P 41c1bfb87491488db8b4877520e5ac60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc188c63638c472e899fcfa41e747aad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41049 } } peers { permanent_uuid: "90534941bd5549558e508bdb29ac8265" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "41c1bfb87491488db8b4877520e5ac60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 } }
I20260925 04:38:57.586611 2649 raft_consensus.cc:374] T 5aaf797144b043008f0f58c79b4e081c P 41c1bfb87491488db8b4877520e5ac60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:38:57.586640 2649 raft_consensus.cc:729] T 5aaf797144b043008f0f58c79b4e081c P 41c1bfb87491488db8b4877520e5ac60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c1bfb87491488db8b4877520e5ac60, State: Initialized, Role: FOLLOWER
I20260925 04:38:57.586767 2649 consensus_queue.cc:260] T 5aaf797144b043008f0f58c79b4e081c P 41c1bfb87491488db8b4877520e5ac60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc188c63638c472e899fcfa41e747aad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41049 } } peers { permanent_uuid: "90534941bd5549558e508bdb29ac8265" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "41c1bfb87491488db8b4877520e5ac60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 } }
I20260925 04:38:57.587006 2649 ts_tablet_manager.cc:1428] T 5aaf797144b043008f0f58c79b4e081c P 41c1bfb87491488db8b4877520e5ac60: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 04:38:57.589038 2699 ts_tablet_manager.cc:1905] T 626b1f46eca84651875ead9fd8e747f1 P 41c1bfb87491488db8b4877520e5ac60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:38:57.592053 2699 ts_tablet_manager.cc:1918] T 626b1f46eca84651875ead9fd8e747f1 P 41c1bfb87491488db8b4877520e5ac60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 04:38:57.592154 2699 log.cc:1196] T 626b1f46eca84651875ead9fd8e747f1 P 41c1bfb87491488db8b4877520e5ac60: Deleting WAL directory at /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/ts-2/wal/wals/626b1f46eca84651875ead9fd8e747f1
I20260925 04:38:57.592393 2699 ts_tablet_manager.cc:1939] T 626b1f46eca84651875ead9fd8e747f1 P 41c1bfb87491488db8b4877520e5ac60: Deleting consensus metadata
I20260925 04:38:57.593583 2126 catalog_manager.cc:4841] TS 41c1bfb87491488db8b4877520e5ac60 (127.0.98.3:38467): tablet 626b1f46eca84651875ead9fd8e747f1 (table test-table [id=c51d7b60229440a6a77d1a4d8beff158]) successfully deleted
I20260925 04:38:57.657627 2655 raft_consensus.cc:482] T 5aaf797144b043008f0f58c79b4e081c P fc188c63638c472e899fcfa41e747aad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:38:57.657795 2655 raft_consensus.cc:504] T 5aaf797144b043008f0f58c79b4e081c P fc188c63638c472e899fcfa41e747aad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc188c63638c472e899fcfa41e747aad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41049 } } peers { permanent_uuid: "90534941bd5549558e508bdb29ac8265" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "41c1bfb87491488db8b4877520e5ac60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 } }
I20260925 04:38:57.659075 2655 leader_election.cc:288] T 5aaf797144b043008f0f58c79b4e081c P fc188c63638c472e899fcfa41e747aad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90534941bd5549558e508bdb29ac8265 (127.0.98.2:41491), 41c1bfb87491488db8b4877520e5ac60 (127.0.98.3:38467)
I20260925 04:38:57.663177 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:57.660065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54607 (local address 127.0.98.3:38467)
0925 04:38:57.660175 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:57.660179 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:57.660193 (+ 14us) server_negotiation.cc:407] Connection header received
0925 04:38:57.660313 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:57.660315 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:57.660352 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:57.660410 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:38:57.660654 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:57.661142 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:57.662212 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:57.662614 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:57.662709 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:57.662780 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:57.662970 (+ 190us) server_negotiation.cc:299] Negotiation successful
0925 04:38:57.663040 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 04:38:57.663193 2706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:57.659753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38467 (local address 127.0.98.1:54607)
0925 04:38:57.660110 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:57.660123 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:38:57.660208 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:38:57.660428 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:38:57.660431 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:38:57.660445 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:38:57.660555 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:57.660562 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:57.661244 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:57.661248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:57.662085 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:57.662095 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:57.662748 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:57.662752 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:57.662856 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:57.662876 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 04:38:57.662943 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 04:38:57.663025 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":89,"threads_started":1}
I20260925 04:38:57.663785 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5aaf797144b043008f0f58c79b4e081c" candidate_uuid: "fc188c63638c472e899fcfa41e747aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c1bfb87491488db8b4877520e5ac60" is_pre_election: true
I20260925 04:38:57.663957 2579 raft_consensus.cc:2454] T 5aaf797144b043008f0f58c79b4e081c P 41c1bfb87491488db8b4877520e5ac60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc188c63638c472e899fcfa41e747aad in term 0.
I20260925 04:38:57.664072 2705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:57.659364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41491 (local address 127.0.98.1:59031)
0925 04:38:57.659673 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:57.659690 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:38:57.659816 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:38:57.660197 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:38:57.660201 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:38:57.660221 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:38:57.660349 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:57.660356 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:57.661457 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:57.661460 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:57.662635 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:57.662647 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:57.663658 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:57.663661 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:57.663759 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:57.663781 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 04:38:57.663855 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 04:38:57.663929 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":82,"threads_started":1}
I20260925 04:38:57.664083 2671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:57.659473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59031 (local address 127.0.98.2:41491)
0925 04:38:57.659635 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:57.659640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:57.659779 (+ 139us) server_negotiation.cc:407] Connection header received
0925 04:38:57.659926 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:57.659930 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:57.659982 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:57.660072 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:38:57.660498 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:57.661283 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:57.662791 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:57.663372 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:57.663469 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:57.663531 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:57.663879 (+ 348us) server_negotiation.cc:299] Negotiation successful
0925 04:38:57.663968 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 04:38:57.664660 2435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5aaf797144b043008f0f58c79b4e081c" candidate_uuid: "fc188c63638c472e899fcfa41e747aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90534941bd5549558e508bdb29ac8265" is_pre_election: true
I20260925 04:38:57.664829 2435 raft_consensus.cc:2454] T 5aaf797144b043008f0f58c79b4e081c P 90534941bd5549558e508bdb29ac8265 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc188c63638c472e899fcfa41e747aad in term 0.
I20260925 04:38:57.665019 2230 leader_election.cc:302] T 5aaf797144b043008f0f58c79b4e081c P fc188c63638c472e899fcfa41e747aad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41c1bfb87491488db8b4877520e5ac60, fc188c63638c472e899fcfa41e747aad; no voters:
I20260925 04:38:57.665282 2655 raft_consensus.cc:2781] T 5aaf797144b043008f0f58c79b4e081c P fc188c63638c472e899fcfa41e747aad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:38:57.665351 2655 raft_consensus.cc:482] T 5aaf797144b043008f0f58c79b4e081c P fc188c63638c472e899fcfa41e747aad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:38:57.665390 2655 raft_consensus.cc:3037] T 5aaf797144b043008f0f58c79b4e081c P fc188c63638c472e899fcfa41e747aad [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:38:57.666721 2655 raft_consensus.cc:504] T 5aaf797144b043008f0f58c79b4e081c P fc188c63638c472e899fcfa41e747aad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc188c63638c472e899fcfa41e747aad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41049 } } peers { permanent_uuid: "90534941bd5549558e508bdb29ac8265" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "41c1bfb87491488db8b4877520e5ac60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 } }
I20260925 04:38:57.667142 2655 leader_election.cc:288] T 5aaf797144b043008f0f58c79b4e081c P fc188c63638c472e899fcfa41e747aad [CANDIDATE]: Term 1 election: Requested vote from peers 90534941bd5549558e508bdb29ac8265 (127.0.98.2:41491), 41c1bfb87491488db8b4877520e5ac60 (127.0.98.3:38467)
I20260925 04:38:57.667716 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5aaf797144b043008f0f58c79b4e081c" candidate_uuid: "fc188c63638c472e899fcfa41e747aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c1bfb87491488db8b4877520e5ac60"
I20260925 04:38:57.667745 2435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5aaf797144b043008f0f58c79b4e081c" candidate_uuid: "fc188c63638c472e899fcfa41e747aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90534941bd5549558e508bdb29ac8265"
I20260925 04:38:57.667860 2579 raft_consensus.cc:3037] T 5aaf797144b043008f0f58c79b4e081c P 41c1bfb87491488db8b4877520e5ac60 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:38:57.667876 2435 raft_consensus.cc:3037] T 5aaf797144b043008f0f58c79b4e081c P 90534941bd5549558e508bdb29ac8265 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:38:57.668839 2579 raft_consensus.cc:2454] T 5aaf797144b043008f0f58c79b4e081c P 41c1bfb87491488db8b4877520e5ac60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc188c63638c472e899fcfa41e747aad in term 1.
I20260925 04:38:57.669173 2230 leader_election.cc:302] T 5aaf797144b043008f0f58c79b4e081c P fc188c63638c472e899fcfa41e747aad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41c1bfb87491488db8b4877520e5ac60, fc188c63638c472e899fcfa41e747aad; no voters:
I20260925 04:38:57.669257 2435 raft_consensus.cc:2454] T 5aaf797144b043008f0f58c79b4e081c P 90534941bd5549558e508bdb29ac8265 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc188c63638c472e899fcfa41e747aad in term 1.
I20260925 04:38:57.669368 2655 raft_consensus.cc:2781] T 5aaf797144b043008f0f58c79b4e081c P fc188c63638c472e899fcfa41e747aad [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:38:57.669655 2655 raft_consensus.cc:686] T 5aaf797144b043008f0f58c79b4e081c P fc188c63638c472e899fcfa41e747aad [term 1 LEADER]: Becoming Leader. State: Replica: fc188c63638c472e899fcfa41e747aad, State: Running, Role: LEADER
I20260925 04:38:57.669926 2655 consensus_queue.cc:237] T 5aaf797144b043008f0f58c79b4e081c P fc188c63638c472e899fcfa41e747aad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc188c63638c472e899fcfa41e747aad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41049 } } peers { permanent_uuid: "90534941bd5549558e508bdb29ac8265" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "41c1bfb87491488db8b4877520e5ac60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 } }
I20260925 04:38:57.672320 2140 catalog_manager.cc:5495] T 5aaf797144b043008f0f58c79b4e081c P fc188c63638c472e899fcfa41e747aad reported cstate change: term changed from 0 to 1, leader changed from <none> to fc188c63638c472e899fcfa41e747aad (127.0.98.1). New cstate: current_term: 1 leader_uuid: "fc188c63638c472e899fcfa41e747aad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc188c63638c472e899fcfa41e747aad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90534941bd5549558e508bdb29ac8265" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41c1bfb87491488db8b4877520e5ac60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 } health_report { overall_health: UNKNOWN } } }
I20260925 04:38:57.691252 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:57.686394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37376 (local address 127.0.98.1:41049)
0925 04:38:57.686585 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:57.686589 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:57.686608 (+ 19us) server_negotiation.cc:407] Connection header received
0925 04:38:57.686844 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:57.686848 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:57.686918 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:57.687080 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:38:57.687382 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:57.688089 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:57.689471 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:57.689743 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:57.689835 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:57.690245 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:38:57.690780 (+ 535us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:38:57.690882 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:57.691027 (+ 145us) server_negotiation.cc:299] Negotiation successful
0925 04:38:57.691099 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260925 04:38:57.703088 2711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:57.699397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38467 (local address 127.0.98.1:57193)
0925 04:38:57.699736 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:57.699749 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:38:57.699845 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:38:57.700101 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:38:57.700103 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:38:57.700109 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 04:38:57.700216 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:57.700220 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:57.701190 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:57.701192 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:57.701793 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:57.701800 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:57.702345 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:57.702348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:57.702416 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:57.702462 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 04:38:57.702802 (+ 340us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 04:38:57.702809 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 04:38:57.702861 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 04:38:57.702930 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":106,"threads_started":1}
I20260925 04:38:57.703078 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:57.699534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57193 (local address 127.0.98.3:38467)
0925 04:38:57.699705 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:57.699710 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:57.699816 (+ 106us) server_negotiation.cc:407] Connection header received
0925 04:38:57.699936 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:57.699939 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:57.700000 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:57.700082 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:38:57.700310 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:57.701086 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:57.701907 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:57.702231 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:57.702342 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:57.702619 (+ 277us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:38:57.702709 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:38:57.702787 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:57.702878 (+ 91us) server_negotiation.cc:299] Negotiation successful
0925 04:38:57.702937 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 04:38:57.709995 2435 raft_consensus.cc:1261] T 5aaf797144b043008f0f58c79b4e081c P 90534941bd5549558e508bdb29ac8265 [term 1 FOLLOWER]: Refusing update from remote peer fc188c63638c472e899fcfa41e747aad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:38:57.710059 2579 raft_consensus.cc:1261] T 5aaf797144b043008f0f58c79b4e081c P 41c1bfb87491488db8b4877520e5ac60 [term 1 FOLLOWER]: Refusing update from remote peer fc188c63638c472e899fcfa41e747aad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:38:57.710494 2707 consensus_queue.cc:1035] T 5aaf797144b043008f0f58c79b4e081c P fc188c63638c472e899fcfa41e747aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "90534941bd5549558e508bdb29ac8265" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:38:57.710673 2655 consensus_queue.cc:1035] T 5aaf797144b043008f0f58c79b4e081c P fc188c63638c472e899fcfa41e747aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c1bfb87491488db8b4877520e5ac60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:38:57.717433 2677 mvcc.cc:205] Tried to move back new op lower bound from 7333114420055429120 to 7333114419898445824. Current Snapshot: MvccSnapshot[applied={T|T < 7333114420055429120}]
I20260925 04:38:57.731630 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:57.728072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54123 (local address 127.0.98.1:41049)
0925 04:38:57.728175 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:57.728179 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:57.728233 (+ 54us) server_negotiation.cc:407] Connection header received
0925 04:38:57.728385 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:57.728388 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:57.728436 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:57.728498 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:38:57.728760 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:57.729539 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:57.730576 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:57.730839 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:57.730927 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:57.731176 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:38:57.731249 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:38:57.731321 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:57.731409 (+ 88us) server_negotiation.cc:299] Negotiation successful
0925 04:38:57.731458 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 04:38:57.731626 2695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:57.728031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41049 (local address 127.0.98.3:54123)
0925 04:38:57.728181 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:57.728195 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:38:57.728276 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:38:57.728524 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:38:57.728527 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:38:57.728534 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 04:38:57.728629 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:57.728636 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:57.729654 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:57.729657 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:57.730440 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:57.730451 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:57.730941 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:57.730945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:57.731022 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:57.731055 (+ 33us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 04:38:57.731327 (+ 272us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 04:38:57.731334 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 04:38:57.731389 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 04:38:57.731475 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
W20260925 04:38:57.732515 2678 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 04:38:59.291486 2140 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37978:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:38:59.291668 2140 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37978:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:38:59.294215 2140 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3d4badfd4e7f4245a022ad52c0385cce: Sending DeleteTablet for 3 replicas of tablet 5aaf797144b043008f0f58c79b4e081c
I20260925 04:38:59.294844 2279 tablet_service.cc:1514] Processing DeleteTablet for tablet 5aaf797144b043008f0f58c79b4e081c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:38:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37366
I20260925 04:38:59.294914 2419 tablet_service.cc:1514] Processing DeleteTablet for tablet 5aaf797144b043008f0f58c79b4e081c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:38:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41962
I20260925 04:38:59.294904 2559 tablet_service.cc:1514] Processing DeleteTablet for tablet 5aaf797144b043008f0f58c79b4e081c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:38:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36102
I20260925 04:38:59.295302 2740 tablet_replica.cc:330] T 5aaf797144b043008f0f58c79b4e081c P 41c1bfb87491488db8b4877520e5ac60: stopping tablet replica
I20260925 04:38:59.295318 2739 tablet_replica.cc:330] T 5aaf797144b043008f0f58c79b4e081c P 90534941bd5549558e508bdb29ac8265: stopping tablet replica
I20260925 04:38:59.295433 2740 raft_consensus.cc:2229] T 5aaf797144b043008f0f58c79b4e081c P 41c1bfb87491488db8b4877520e5ac60 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:38:59.295442 2739 raft_consensus.cc:2229] T 5aaf797144b043008f0f58c79b4e081c P 90534941bd5549558e508bdb29ac8265 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:38:59.295558 2739 raft_consensus.cc:2258] T 5aaf797144b043008f0f58c79b4e081c P 90534941bd5549558e508bdb29ac8265 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:38:59.295560 2740 raft_consensus.cc:2258] T 5aaf797144b043008f0f58c79b4e081c P 41c1bfb87491488db8b4877520e5ac60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:38:59.296205 2740 ts_tablet_manager.cc:1905] T 5aaf797144b043008f0f58c79b4e081c P 41c1bfb87491488db8b4877520e5ac60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:38:59.296386 2739 ts_tablet_manager.cc:1905] T 5aaf797144b043008f0f58c79b4e081c P 90534941bd5549558e508bdb29ac8265: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:38:59.299096 2740 ts_tablet_manager.cc:1918] T 5aaf797144b043008f0f58c79b4e081c P 41c1bfb87491488db8b4877520e5ac60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 04:38:59.299146 2738 tablet_replica.cc:330] T 5aaf797144b043008f0f58c79b4e081c P fc188c63638c472e899fcfa41e747aad: stopping tablet replica
I20260925 04:38:59.299275 2739 ts_tablet_manager.cc:1918] T 5aaf797144b043008f0f58c79b4e081c P 90534941bd5549558e508bdb29ac8265: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 04:38:59.299336 2740 log.cc:1196] T 5aaf797144b043008f0f58c79b4e081c P 41c1bfb87491488db8b4877520e5ac60: Deleting WAL directory at /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/ts-2/wal/wals/5aaf797144b043008f0f58c79b4e081c
I20260925 04:38:59.299357 2738 raft_consensus.cc:2229] T 5aaf797144b043008f0f58c79b4e081c P fc188c63638c472e899fcfa41e747aad [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:38:59.299475 2739 log.cc:1196] T 5aaf797144b043008f0f58c79b4e081c P 90534941bd5549558e508bdb29ac8265: Deleting WAL directory at /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/ts-1/wal/wals/5aaf797144b043008f0f58c79b4e081c
I20260925 04:38:59.299803 2739 ts_tablet_manager.cc:1939] T 5aaf797144b043008f0f58c79b4e081c P 90534941bd5549558e508bdb29ac8265: Deleting consensus metadata
I20260925 04:38:59.299826 2738 raft_consensus.cc:2258] T 5aaf797144b043008f0f58c79b4e081c P fc188c63638c472e899fcfa41e747aad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:38:59.299876 2740 ts_tablet_manager.cc:1939] T 5aaf797144b043008f0f58c79b4e081c P 41c1bfb87491488db8b4877520e5ac60: Deleting consensus metadata
I20260925 04:38:59.300674 2738 ts_tablet_manager.cc:1905] T 5aaf797144b043008f0f58c79b4e081c P fc188c63638c472e899fcfa41e747aad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:38:59.300745 2126 catalog_manager.cc:4841] TS 41c1bfb87491488db8b4877520e5ac60 (127.0.98.3:38467): tablet 5aaf797144b043008f0f58c79b4e081c (table test-table [id=53673001fc084072ac41d8d52420d59d]) successfully deleted
I20260925 04:38:59.301401 2126 catalog_manager.cc:4841] TS 90534941bd5549558e508bdb29ac8265 (127.0.98.2:41491): tablet 5aaf797144b043008f0f58c79b4e081c (table test-table [id=53673001fc084072ac41d8d52420d59d]) successfully deleted
I20260925 04:38:59.303333 2738 ts_tablet_manager.cc:1918] T 5aaf797144b043008f0f58c79b4e081c P fc188c63638c472e899fcfa41e747aad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 04:38:59.303398 2738 log.cc:1196] T 5aaf797144b043008f0f58c79b4e081c P fc188c63638c472e899fcfa41e747aad: Deleting WAL directory at /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/ts-0/wal/wals/5aaf797144b043008f0f58c79b4e081c
I20260925 04:38:59.303676 2738 ts_tablet_manager.cc:1939] T 5aaf797144b043008f0f58c79b4e081c P fc188c63638c472e899fcfa41e747aad: Deleting consensus metadata
I20260925 04:38:59.304356 2128 catalog_manager.cc:4841] TS fc188c63638c472e899fcfa41e747aad (127.0.98.1:41049): tablet 5aaf797144b043008f0f58c79b4e081c (table test-table [id=53673001fc084072ac41d8d52420d59d]) successfully deleted
Sep 25 04:38:59 dist-test-slave-dc95 krb5kdc[2097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311139, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 04:38:59 dist-test-slave-dc95 krb5kdc[2097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311139, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:38:59.324128 2748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:59.314902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48894 (local address 127.0.98.62:38605)
0925 04:38:59.315173 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:59.315180 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:59.315199 (+ 19us) server_negotiation.cc:407] Connection header received
0925 04:38:59.315350 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:59.315353 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:59.315405 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:59.315498 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:59.316343 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:59.316931 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:59.317888 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:59.318220 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:59.318322 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:59.320972 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:38:59.320976 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:38:59.320978 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:38:59.321010 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:38:59.322402 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:59.322813 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:59.322817 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:59.322818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:59.322900 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:59.323191 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:59.323194 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:59.323196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:59.323438 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:38:59.323566 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:59.323826 (+ 260us) server_negotiation.cc:299] Negotiation successful
0925 04:38:59.323954 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":93,"threads_started":1}
I20260925 04:38:59.326663 2140 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48894:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:38:59.327080 2140 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:38:59.332130 2279 tablet_service.cc:1467] Processing CreateTablet for tablet adc1e1ad4fc148a2bcc07f869ef08f64 (DEFAULT_TABLE table=test-table [id=6025633970304f638adc4688a3e1b6b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:38:59.332465 2279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adc1e1ad4fc148a2bcc07f869ef08f64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:38:59.332410 2419 tablet_service.cc:1467] Processing CreateTablet for tablet adc1e1ad4fc148a2bcc07f869ef08f64 (DEFAULT_TABLE table=test-table [id=6025633970304f638adc4688a3e1b6b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:38:59.332674 2419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adc1e1ad4fc148a2bcc07f869ef08f64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:38:59.333997 2559 tablet_service.cc:1467] Processing CreateTablet for tablet adc1e1ad4fc148a2bcc07f869ef08f64 (DEFAULT_TABLE table=test-table [id=6025633970304f638adc4688a3e1b6b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:38:59.334290 2559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adc1e1ad4fc148a2bcc07f869ef08f64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:38:59.334978 2749 tablet_bootstrap.cc:492] T adc1e1ad4fc148a2bcc07f869ef08f64 P fc188c63638c472e899fcfa41e747aad: Bootstrap starting.
I20260925 04:38:59.335816 2750 tablet_bootstrap.cc:492] T adc1e1ad4fc148a2bcc07f869ef08f64 P 90534941bd5549558e508bdb29ac8265: Bootstrap starting.
I20260925 04:38:59.335865 2749 tablet_bootstrap.cc:654] T adc1e1ad4fc148a2bcc07f869ef08f64 P fc188c63638c472e899fcfa41e747aad: Neither blocks nor log segments found. Creating new log.
I20260925 04:38:59.336633 2750 tablet_bootstrap.cc:654] T adc1e1ad4fc148a2bcc07f869ef08f64 P 90534941bd5549558e508bdb29ac8265: Neither blocks nor log segments found. Creating new log.
I20260925 04:38:59.337105 2749 tablet_bootstrap.cc:492] T adc1e1ad4fc148a2bcc07f869ef08f64 P fc188c63638c472e899fcfa41e747aad: No bootstrap required, opened a new log
I20260925 04:38:59.337198 2749 ts_tablet_manager.cc:1397] T adc1e1ad4fc148a2bcc07f869ef08f64 P fc188c63638c472e899fcfa41e747aad: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 04:38:59.337587 2749 raft_consensus.cc:348] T adc1e1ad4fc148a2bcc07f869ef08f64 P fc188c63638c472e899fcfa41e747aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc188c63638c472e899fcfa41e747aad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41049 } } peers { permanent_uuid: "90534941bd5549558e508bdb29ac8265" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "41c1bfb87491488db8b4877520e5ac60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 } }
I20260925 04:38:59.337708 2749 raft_consensus.cc:374] T adc1e1ad4fc148a2bcc07f869ef08f64 P fc188c63638c472e899fcfa41e747aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:38:59.337746 2749 raft_consensus.cc:729] T adc1e1ad4fc148a2bcc07f869ef08f64 P fc188c63638c472e899fcfa41e747aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc188c63638c472e899fcfa41e747aad, State: Initialized, Role: FOLLOWER
I20260925 04:38:59.337904 2750 tablet_bootstrap.cc:492] T adc1e1ad4fc148a2bcc07f869ef08f64 P 90534941bd5549558e508bdb29ac8265: No bootstrap required, opened a new log
I20260925 04:38:59.337882 2749 consensus_queue.cc:260] T adc1e1ad4fc148a2bcc07f869ef08f64 P fc188c63638c472e899fcfa41e747aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc188c63638c472e899fcfa41e747aad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41049 } } peers { permanent_uuid: "90534941bd5549558e508bdb29ac8265" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "41c1bfb87491488db8b4877520e5ac60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 } }
I20260925 04:38:59.338030 2750 ts_tablet_manager.cc:1397] T adc1e1ad4fc148a2bcc07f869ef08f64 P 90534941bd5549558e508bdb29ac8265: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 04:38:59.338148 2749 ts_tablet_manager.cc:1428] T adc1e1ad4fc148a2bcc07f869ef08f64 P fc188c63638c472e899fcfa41e747aad: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 04:38:59.338539 2752 tablet_bootstrap.cc:492] T adc1e1ad4fc148a2bcc07f869ef08f64 P 41c1bfb87491488db8b4877520e5ac60: Bootstrap starting.
I20260925 04:38:59.338594 2750 raft_consensus.cc:348] T adc1e1ad4fc148a2bcc07f869ef08f64 P 90534941bd5549558e508bdb29ac8265 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc188c63638c472e899fcfa41e747aad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41049 } } peers { permanent_uuid: "90534941bd5549558e508bdb29ac8265" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "41c1bfb87491488db8b4877520e5ac60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 } }
I20260925 04:38:59.338708 2750 raft_consensus.cc:374] T adc1e1ad4fc148a2bcc07f869ef08f64 P 90534941bd5549558e508bdb29ac8265 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:38:59.338776 2750 raft_consensus.cc:729] T adc1e1ad4fc148a2bcc07f869ef08f64 P 90534941bd5549558e508bdb29ac8265 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90534941bd5549558e508bdb29ac8265, State: Initialized, Role: FOLLOWER
I20260925 04:38:59.338930 2750 consensus_queue.cc:260] T adc1e1ad4fc148a2bcc07f869ef08f64 P 90534941bd5549558e508bdb29ac8265 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc188c63638c472e899fcfa41e747aad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41049 } } peers { permanent_uuid: "90534941bd5549558e508bdb29ac8265" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "41c1bfb87491488db8b4877520e5ac60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 } }
I20260925 04:38:59.339327 2750 ts_tablet_manager.cc:1428] T adc1e1ad4fc148a2bcc07f869ef08f64 P 90534941bd5549558e508bdb29ac8265: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 04:38:59.339414 2752 tablet_bootstrap.cc:654] T adc1e1ad4fc148a2bcc07f869ef08f64 P 41c1bfb87491488db8b4877520e5ac60: Neither blocks nor log segments found. Creating new log.
I20260925 04:38:59.341107 2752 tablet_bootstrap.cc:492] T adc1e1ad4fc148a2bcc07f869ef08f64 P 41c1bfb87491488db8b4877520e5ac60: No bootstrap required, opened a new log
I20260925 04:38:59.341207 2752 ts_tablet_manager.cc:1397] T adc1e1ad4fc148a2bcc07f869ef08f64 P 41c1bfb87491488db8b4877520e5ac60: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 04:38:59.341679 2752 raft_consensus.cc:348] T adc1e1ad4fc148a2bcc07f869ef08f64 P 41c1bfb87491488db8b4877520e5ac60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc188c63638c472e899fcfa41e747aad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41049 } } peers { permanent_uuid: "90534941bd5549558e508bdb29ac8265" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "41c1bfb87491488db8b4877520e5ac60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 } }
I20260925 04:38:59.341797 2752 raft_consensus.cc:374] T adc1e1ad4fc148a2bcc07f869ef08f64 P 41c1bfb87491488db8b4877520e5ac60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:38:59.341836 2752 raft_consensus.cc:729] T adc1e1ad4fc148a2bcc07f869ef08f64 P 41c1bfb87491488db8b4877520e5ac60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c1bfb87491488db8b4877520e5ac60, State: Initialized, Role: FOLLOWER
I20260925 04:38:59.341961 2752 consensus_queue.cc:260] T adc1e1ad4fc148a2bcc07f869ef08f64 P 41c1bfb87491488db8b4877520e5ac60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc188c63638c472e899fcfa41e747aad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41049 } } peers { permanent_uuid: "90534941bd5549558e508bdb29ac8265" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "41c1bfb87491488db8b4877520e5ac60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 } }
I20260925 04:38:59.342202 2752 ts_tablet_manager.cc:1428] T adc1e1ad4fc148a2bcc07f869ef08f64 P 41c1bfb87491488db8b4877520e5ac60: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 04:38:59.379127 2733 raft_consensus.cc:482] T adc1e1ad4fc148a2bcc07f869ef08f64 P 41c1bfb87491488db8b4877520e5ac60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:38:59.379292 2733 raft_consensus.cc:504] T adc1e1ad4fc148a2bcc07f869ef08f64 P 41c1bfb87491488db8b4877520e5ac60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc188c63638c472e899fcfa41e747aad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41049 } } peers { permanent_uuid: "90534941bd5549558e508bdb29ac8265" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "41c1bfb87491488db8b4877520e5ac60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 } }
I20260925 04:38:59.379746 2733 leader_election.cc:288] T adc1e1ad4fc148a2bcc07f869ef08f64 P 41c1bfb87491488db8b4877520e5ac60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc188c63638c472e899fcfa41e747aad (127.0.98.1:41049), 90534941bd5549558e508bdb29ac8265 (127.0.98.2:41491)
I20260925 04:38:59.380215 2299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adc1e1ad4fc148a2bcc07f869ef08f64" candidate_uuid: "41c1bfb87491488db8b4877520e5ac60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc188c63638c472e899fcfa41e747aad" is_pre_election: true
I20260925 04:38:59.380225 2439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adc1e1ad4fc148a2bcc07f869ef08f64" candidate_uuid: "41c1bfb87491488db8b4877520e5ac60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90534941bd5549558e508bdb29ac8265" is_pre_election: true
I20260925 04:38:59.380383 2299 raft_consensus.cc:2454] T adc1e1ad4fc148a2bcc07f869ef08f64 P fc188c63638c472e899fcfa41e747aad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41c1bfb87491488db8b4877520e5ac60 in term 0.
I20260925 04:38:59.380383 2439 raft_consensus.cc:2454] T adc1e1ad4fc148a2bcc07f869ef08f64 P 90534941bd5549558e508bdb29ac8265 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41c1bfb87491488db8b4877520e5ac60 in term 0.
I20260925 04:38:59.380693 2512 leader_election.cc:302] T adc1e1ad4fc148a2bcc07f869ef08f64 P 41c1bfb87491488db8b4877520e5ac60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41c1bfb87491488db8b4877520e5ac60, fc188c63638c472e899fcfa41e747aad; no voters:
I20260925 04:38:59.380890 2733 raft_consensus.cc:2781] T adc1e1ad4fc148a2bcc07f869ef08f64 P 41c1bfb87491488db8b4877520e5ac60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:38:59.380947 2733 raft_consensus.cc:482] T adc1e1ad4fc148a2bcc07f869ef08f64 P 41c1bfb87491488db8b4877520e5ac60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:38:59.380975 2733 raft_consensus.cc:3037] T adc1e1ad4fc148a2bcc07f869ef08f64 P 41c1bfb87491488db8b4877520e5ac60 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:38:59.381973 2733 raft_consensus.cc:504] T adc1e1ad4fc148a2bcc07f869ef08f64 P 41c1bfb87491488db8b4877520e5ac60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc188c63638c472e899fcfa41e747aad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41049 } } peers { permanent_uuid: "90534941bd5549558e508bdb29ac8265" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "41c1bfb87491488db8b4877520e5ac60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 } }
I20260925 04:38:59.382311 2733 leader_election.cc:288] T adc1e1ad4fc148a2bcc07f869ef08f64 P 41c1bfb87491488db8b4877520e5ac60 [CANDIDATE]: Term 1 election: Requested vote from peers fc188c63638c472e899fcfa41e747aad (127.0.98.1:41049), 90534941bd5549558e508bdb29ac8265 (127.0.98.2:41491)
I20260925 04:38:59.382676 2439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adc1e1ad4fc148a2bcc07f869ef08f64" candidate_uuid: "41c1bfb87491488db8b4877520e5ac60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90534941bd5549558e508bdb29ac8265"
I20260925 04:38:59.382676 2299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adc1e1ad4fc148a2bcc07f869ef08f64" candidate_uuid: "41c1bfb87491488db8b4877520e5ac60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc188c63638c472e899fcfa41e747aad"
I20260925 04:38:59.382804 2439 raft_consensus.cc:3037] T adc1e1ad4fc148a2bcc07f869ef08f64 P 90534941bd5549558e508bdb29ac8265 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:38:59.382808 2299 raft_consensus.cc:3037] T adc1e1ad4fc148a2bcc07f869ef08f64 P fc188c63638c472e899fcfa41e747aad [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:38:59.383817 2439 raft_consensus.cc:2454] T adc1e1ad4fc148a2bcc07f869ef08f64 P 90534941bd5549558e508bdb29ac8265 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41c1bfb87491488db8b4877520e5ac60 in term 1.
I20260925 04:38:59.383831 2299 raft_consensus.cc:2454] T adc1e1ad4fc148a2bcc07f869ef08f64 P fc188c63638c472e899fcfa41e747aad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41c1bfb87491488db8b4877520e5ac60 in term 1.
I20260925 04:38:59.384109 2510 leader_election.cc:302] T adc1e1ad4fc148a2bcc07f869ef08f64 P 41c1bfb87491488db8b4877520e5ac60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41c1bfb87491488db8b4877520e5ac60, 90534941bd5549558e508bdb29ac8265; no voters:
I20260925 04:38:59.384277 2733 raft_consensus.cc:2781] T adc1e1ad4fc148a2bcc07f869ef08f64 P 41c1bfb87491488db8b4877520e5ac60 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:38:59.384366 2733 raft_consensus.cc:686] T adc1e1ad4fc148a2bcc07f869ef08f64 P 41c1bfb87491488db8b4877520e5ac60 [term 1 LEADER]: Becoming Leader. State: Replica: 41c1bfb87491488db8b4877520e5ac60, State: Running, Role: LEADER
I20260925 04:38:59.384542 2733 consensus_queue.cc:237] T adc1e1ad4fc148a2bcc07f869ef08f64 P 41c1bfb87491488db8b4877520e5ac60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc188c63638c472e899fcfa41e747aad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41049 } } peers { permanent_uuid: "90534941bd5549558e508bdb29ac8265" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "41c1bfb87491488db8b4877520e5ac60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 } }
I20260925 04:38:59.386200 2141 catalog_manager.cc:5495] T adc1e1ad4fc148a2bcc07f869ef08f64 P 41c1bfb87491488db8b4877520e5ac60 reported cstate change: term changed from 0 to 1, leader changed from <none> to 41c1bfb87491488db8b4877520e5ac60 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "41c1bfb87491488db8b4877520e5ac60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc188c63638c472e899fcfa41e747aad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90534941bd5549558e508bdb29ac8265" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41c1bfb87491488db8b4877520e5ac60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 } health_report { overall_health: HEALTHY } } }
I20260925 04:38:59.412524 2758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:59.409317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59412 (local address 127.0.98.3:38467)
0925 04:38:59.409556 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:59.409562 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:59.409574 (+ 12us) server_negotiation.cc:407] Connection header received
0925 04:38:59.409623 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:59.409625 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:59.409668 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:59.409742 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:38:59.409956 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:59.410475 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:59.411390 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:59.411657 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:59.411824 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:59.412035 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:38:59.412108 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:38:59.412170 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:59.412292 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 04:38:59.412347 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":93,"threads_started":1}
I20260925 04:38:59.421516 2761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:59.418418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38467 (local address 127.0.98.3:39677)
0925 04:38:59.418672 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:59.418688 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:38:59.418799 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:38:59.418995 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:38:59.418998 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:38:59.419004 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 04:38:59.419109 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:59.419114 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:59.419891 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:59.419894 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:59.420481 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:59.420489 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:59.420881 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:59.420884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:59.420986 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:59.421007 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 04:38:59.421240 (+ 233us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 04:38:59.421245 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 04:38:59.421297 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 04:38:59.421369 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":18,"thread_start_us":109,"threads_started":1}
I20260925 04:38:59.421517 2758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:59.418584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39677 (local address 127.0.98.3:38467)
0925 04:38:59.418739 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:59.418744 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:59.418756 (+ 12us) server_negotiation.cc:407] Connection header received
0925 04:38:59.418883 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:59.418886 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:59.418924 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:59.418980 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:38:59.419193 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:59.419796 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:59.420579 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:59.420787 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:59.420869 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:59.421099 (+ 230us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:38:59.421167 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:38:59.421223 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:59.421315 (+ 92us) server_negotiation.cc:299] Negotiation successful
0925 04:38:59.421358 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":97}
I20260925 04:38:59.426921 2299 raft_consensus.cc:1261] T adc1e1ad4fc148a2bcc07f869ef08f64 P fc188c63638c472e899fcfa41e747aad [term 1 FOLLOWER]: Refusing update from remote peer 41c1bfb87491488db8b4877520e5ac60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:38:59.427014 2439 raft_consensus.cc:1261] T adc1e1ad4fc148a2bcc07f869ef08f64 P 90534941bd5549558e508bdb29ac8265 [term 1 FOLLOWER]: Refusing update from remote peer 41c1bfb87491488db8b4877520e5ac60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:38:59.427280 2719 consensus_queue.cc:1035] T adc1e1ad4fc148a2bcc07f869ef08f64 P 41c1bfb87491488db8b4877520e5ac60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc188c63638c472e899fcfa41e747aad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:38:59.427425 2716 consensus_queue.cc:1035] T adc1e1ad4fc148a2bcc07f869ef08f64 P 41c1bfb87491488db8b4877520e5ac60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90534941bd5549558e508bdb29ac8265" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:38:59.460975 2141 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:38:59.461135 2141 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:38:59.463321 2141 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3d4badfd4e7f4245a022ad52c0385cce: Sending DeleteTablet for 3 replicas of tablet adc1e1ad4fc148a2bcc07f869ef08f64
I20260925 04:38:59.464079 2279 tablet_service.cc:1514] Processing DeleteTablet for tablet adc1e1ad4fc148a2bcc07f869ef08f64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:38:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37366
I20260925 04:38:59.464079 2419 tablet_service.cc:1514] Processing DeleteTablet for tablet adc1e1ad4fc148a2bcc07f869ef08f64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:38:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41962
I20260925 04:38:59.464299 2738 tablet_replica.cc:330] T adc1e1ad4fc148a2bcc07f869ef08f64 P fc188c63638c472e899fcfa41e747aad: stopping tablet replica
I20260925 04:38:59.464435 2738 raft_consensus.cc:2229] T adc1e1ad4fc148a2bcc07f869ef08f64 P fc188c63638c472e899fcfa41e747aad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:38:59.464449 2739 tablet_replica.cc:330] T adc1e1ad4fc148a2bcc07f869ef08f64 P 90534941bd5549558e508bdb29ac8265: stopping tablet replica
I20260925 04:38:59.464437 2559 tablet_service.cc:1514] Processing DeleteTablet for tablet adc1e1ad4fc148a2bcc07f869ef08f64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:38:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36102
I20260925 04:38:59.464560 2738 raft_consensus.cc:2258] T adc1e1ad4fc148a2bcc07f869ef08f64 P fc188c63638c472e899fcfa41e747aad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:38:59.464560 2739 raft_consensus.cc:2229] T adc1e1ad4fc148a2bcc07f869ef08f64 P 90534941bd5549558e508bdb29ac8265 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:38:59.464608 2740 tablet_replica.cc:330] T adc1e1ad4fc148a2bcc07f869ef08f64 P 41c1bfb87491488db8b4877520e5ac60: stopping tablet replica
I20260925 04:38:59.464658 2739 raft_consensus.cc:2258] T adc1e1ad4fc148a2bcc07f869ef08f64 P 90534941bd5549558e508bdb29ac8265 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:38:59.464716 2740 raft_consensus.cc:2229] T adc1e1ad4fc148a2bcc07f869ef08f64 P 41c1bfb87491488db8b4877520e5ac60 [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:38:59.465037 2740 raft_consensus.cc:2258] T adc1e1ad4fc148a2bcc07f869ef08f64 P 41c1bfb87491488db8b4877520e5ac60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:38:59.465334 2738 ts_tablet_manager.cc:1905] T adc1e1ad4fc148a2bcc07f869ef08f64 P fc188c63638c472e899fcfa41e747aad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:38:59.465548 2739 ts_tablet_manager.cc:1905] T adc1e1ad4fc148a2bcc07f869ef08f64 P 90534941bd5549558e508bdb29ac8265: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:38:59.465803 2740 ts_tablet_manager.cc:1905] T adc1e1ad4fc148a2bcc07f869ef08f64 P 41c1bfb87491488db8b4877520e5ac60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:38:59.468067 2738 ts_tablet_manager.cc:1918] T adc1e1ad4fc148a2bcc07f869ef08f64 P fc188c63638c472e899fcfa41e747aad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 04:38:59.468150 2738 log.cc:1196] T adc1e1ad4fc148a2bcc07f869ef08f64 P fc188c63638c472e899fcfa41e747aad: Deleting WAL directory at /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/ts-0/wal/wals/adc1e1ad4fc148a2bcc07f869ef08f64
I20260925 04:38:59.468134 2739 ts_tablet_manager.cc:1918] T adc1e1ad4fc148a2bcc07f869ef08f64 P 90534941bd5549558e508bdb29ac8265: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 04:38:59.468207 2739 log.cc:1196] T adc1e1ad4fc148a2bcc07f869ef08f64 P 90534941bd5549558e508bdb29ac8265: Deleting WAL directory at /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/ts-1/wal/wals/adc1e1ad4fc148a2bcc07f869ef08f64
I20260925 04:38:59.468349 2740 ts_tablet_manager.cc:1918] T adc1e1ad4fc148a2bcc07f869ef08f64 P 41c1bfb87491488db8b4877520e5ac60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 04:38:59.468385 2738 ts_tablet_manager.cc:1939] T adc1e1ad4fc148a2bcc07f869ef08f64 P fc188c63638c472e899fcfa41e747aad: Deleting consensus metadata
I20260925 04:38:59.468436 2739 ts_tablet_manager.cc:1939] T adc1e1ad4fc148a2bcc07f869ef08f64 P 90534941bd5549558e508bdb29ac8265: Deleting consensus metadata
I20260925 04:38:59.468429 2740 log.cc:1196] T adc1e1ad4fc148a2bcc07f869ef08f64 P 41c1bfb87491488db8b4877520e5ac60: Deleting WAL directory at /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/ts-2/wal/wals/adc1e1ad4fc148a2bcc07f869ef08f64
I20260925 04:38:59.468641 2740 ts_tablet_manager.cc:1939] T adc1e1ad4fc148a2bcc07f869ef08f64 P 41c1bfb87491488db8b4877520e5ac60: Deleting consensus metadata
I20260925 04:38:59.469326 2128 catalog_manager.cc:4841] TS fc188c63638c472e899fcfa41e747aad (127.0.98.1:41049): tablet adc1e1ad4fc148a2bcc07f869ef08f64 (table test-table [id=6025633970304f638adc4688a3e1b6b8]) successfully deleted
I20260925 04:38:59.469417 2141 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48894:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 04:38:59.469326 2126 catalog_manager.cc:4841] TS 41c1bfb87491488db8b4877520e5ac60 (127.0.98.3:38467): tablet adc1e1ad4fc148a2bcc07f869ef08f64 (table test-table [id=6025633970304f638adc4688a3e1b6b8]) successfully deleted
I20260925 04:38:59.469719 2126 catalog_manager.cc:4841] TS 90534941bd5549558e508bdb29ac8265 (127.0.98.2:41491): tablet adc1e1ad4fc148a2bcc07f869ef08f64 (table test-table [id=6025633970304f638adc4688a3e1b6b8]) successfully deleted
W20260925 04:38:59.469899 2141 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:38:59.475273 2559 tablet_service.cc:1467] Processing CreateTablet for tablet 5dcbb16c911b4a30b01af325083e9730 (DEFAULT_TABLE table=test-table [id=777feaf30c6240d4aeda0269b8d6a29c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:38:59.475263 2279 tablet_service.cc:1467] Processing CreateTablet for tablet 5dcbb16c911b4a30b01af325083e9730 (DEFAULT_TABLE table=test-table [id=777feaf30c6240d4aeda0269b8d6a29c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:38:59.475365 2419 tablet_service.cc:1467] Processing CreateTablet for tablet 5dcbb16c911b4a30b01af325083e9730 (DEFAULT_TABLE table=test-table [id=777feaf30c6240d4aeda0269b8d6a29c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:38:59.475627 2419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dcbb16c911b4a30b01af325083e9730. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:38:59.475625 2279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dcbb16c911b4a30b01af325083e9730. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:38:59.475647 2559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dcbb16c911b4a30b01af325083e9730. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:38:59.478142 2750 tablet_bootstrap.cc:492] T 5dcbb16c911b4a30b01af325083e9730 P 90534941bd5549558e508bdb29ac8265: Bootstrap starting.
I20260925 04:38:59.478142 2749 tablet_bootstrap.cc:492] T 5dcbb16c911b4a30b01af325083e9730 P fc188c63638c472e899fcfa41e747aad: Bootstrap starting.
I20260925 04:38:59.478238 2752 tablet_bootstrap.cc:492] T 5dcbb16c911b4a30b01af325083e9730 P 41c1bfb87491488db8b4877520e5ac60: Bootstrap starting.
I20260925 04:38:59.479106 2752 tablet_bootstrap.cc:654] T 5dcbb16c911b4a30b01af325083e9730 P 41c1bfb87491488db8b4877520e5ac60: Neither blocks nor log segments found. Creating new log.
I20260925 04:38:59.479142 2749 tablet_bootstrap.cc:654] T 5dcbb16c911b4a30b01af325083e9730 P fc188c63638c472e899fcfa41e747aad: Neither blocks nor log segments found. Creating new log.
I20260925 04:38:59.479142 2750 tablet_bootstrap.cc:654] T 5dcbb16c911b4a30b01af325083e9730 P 90534941bd5549558e508bdb29ac8265: Neither blocks nor log segments found. Creating new log.
I20260925 04:38:59.480561 2750 tablet_bootstrap.cc:492] T 5dcbb16c911b4a30b01af325083e9730 P 90534941bd5549558e508bdb29ac8265: No bootstrap required, opened a new log
I20260925 04:38:59.480664 2750 ts_tablet_manager.cc:1397] T 5dcbb16c911b4a30b01af325083e9730 P 90534941bd5549558e508bdb29ac8265: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 04:38:59.480815 2749 tablet_bootstrap.cc:492] T 5dcbb16c911b4a30b01af325083e9730 P fc188c63638c472e899fcfa41e747aad: No bootstrap required, opened a new log
I20260925 04:38:59.480893 2749 ts_tablet_manager.cc:1397] T 5dcbb16c911b4a30b01af325083e9730 P fc188c63638c472e899fcfa41e747aad: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 04:38:59.481115 2750 raft_consensus.cc:348] T 5dcbb16c911b4a30b01af325083e9730 P 90534941bd5549558e508bdb29ac8265 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc188c63638c472e899fcfa41e747aad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41049 } } peers { permanent_uuid: "90534941bd5549558e508bdb29ac8265" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "41c1bfb87491488db8b4877520e5ac60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 } }
I20260925 04:38:59.481233 2750 raft_consensus.cc:374] T 5dcbb16c911b4a30b01af325083e9730 P 90534941bd5549558e508bdb29ac8265 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:38:59.481267 2750 raft_consensus.cc:729] T 5dcbb16c911b4a30b01af325083e9730 P 90534941bd5549558e508bdb29ac8265 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90534941bd5549558e508bdb29ac8265, State: Initialized, Role: FOLLOWER
I20260925 04:38:59.481252 2749 raft_consensus.cc:348] T 5dcbb16c911b4a30b01af325083e9730 P fc188c63638c472e899fcfa41e747aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc188c63638c472e899fcfa41e747aad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41049 } } peers { permanent_uuid: "90534941bd5549558e508bdb29ac8265" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "41c1bfb87491488db8b4877520e5ac60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 } }
I20260925 04:38:59.481348 2749 raft_consensus.cc:374] T 5dcbb16c911b4a30b01af325083e9730 P fc188c63638c472e899fcfa41e747aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:38:59.481379 2749 raft_consensus.cc:729] T 5dcbb16c911b4a30b01af325083e9730 P fc188c63638c472e899fcfa41e747aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc188c63638c472e899fcfa41e747aad, State: Initialized, Role: FOLLOWER
I20260925 04:38:59.481410 2750 consensus_queue.cc:260] T 5dcbb16c911b4a30b01af325083e9730 P 90534941bd5549558e508bdb29ac8265 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc188c63638c472e899fcfa41e747aad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41049 } } peers { permanent_uuid: "90534941bd5549558e508bdb29ac8265" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "41c1bfb87491488db8b4877520e5ac60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 } }
I20260925 04:38:59.481485 2749 consensus_queue.cc:260] T 5dcbb16c911b4a30b01af325083e9730 P fc188c63638c472e899fcfa41e747aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc188c63638c472e899fcfa41e747aad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41049 } } peers { permanent_uuid: "90534941bd5549558e508bdb29ac8265" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "41c1bfb87491488db8b4877520e5ac60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 } }
I20260925 04:38:59.481681 2750 ts_tablet_manager.cc:1428] T 5dcbb16c911b4a30b01af325083e9730 P 90534941bd5549558e508bdb29ac8265: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 04:38:59.481724 2749 ts_tablet_manager.cc:1428] T 5dcbb16c911b4a30b01af325083e9730 P fc188c63638c472e899fcfa41e747aad: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 04:38:59.483783 2752 tablet_bootstrap.cc:492] T 5dcbb16c911b4a30b01af325083e9730 P 41c1bfb87491488db8b4877520e5ac60: No bootstrap required, opened a new log
I20260925 04:38:59.484026 2752 ts_tablet_manager.cc:1397] T 5dcbb16c911b4a30b01af325083e9730 P 41c1bfb87491488db8b4877520e5ac60: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260925 04:38:59.484529 2752 raft_consensus.cc:348] T 5dcbb16c911b4a30b01af325083e9730 P 41c1bfb87491488db8b4877520e5ac60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc188c63638c472e899fcfa41e747aad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41049 } } peers { permanent_uuid: "90534941bd5549558e508bdb29ac8265" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "41c1bfb87491488db8b4877520e5ac60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 } }
I20260925 04:38:59.484679 2752 raft_consensus.cc:374] T 5dcbb16c911b4a30b01af325083e9730 P 41c1bfb87491488db8b4877520e5ac60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:38:59.484714 2752 raft_consensus.cc:729] T 5dcbb16c911b4a30b01af325083e9730 P 41c1bfb87491488db8b4877520e5ac60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c1bfb87491488db8b4877520e5ac60, State: Initialized, Role: FOLLOWER
I20260925 04:38:59.484965 2752 consensus_queue.cc:260] T 5dcbb16c911b4a30b01af325083e9730 P 41c1bfb87491488db8b4877520e5ac60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc188c63638c472e899fcfa41e747aad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41049 } } peers { permanent_uuid: "90534941bd5549558e508bdb29ac8265" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "41c1bfb87491488db8b4877520e5ac60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 } }
I20260925 04:38:59.485301 2752 ts_tablet_manager.cc:1428] T 5dcbb16c911b4a30b01af325083e9730 P 41c1bfb87491488db8b4877520e5ac60: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 04:38:59.505820 2731 raft_consensus.cc:482] T 5dcbb16c911b4a30b01af325083e9730 P fc188c63638c472e899fcfa41e747aad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:38:59.505986 2731 raft_consensus.cc:504] T 5dcbb16c911b4a30b01af325083e9730 P fc188c63638c472e899fcfa41e747aad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc188c63638c472e899fcfa41e747aad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41049 } } peers { permanent_uuid: "90534941bd5549558e508bdb29ac8265" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "41c1bfb87491488db8b4877520e5ac60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 } }
I20260925 04:38:59.506433 2731 leader_election.cc:288] T 5dcbb16c911b4a30b01af325083e9730 P fc188c63638c472e899fcfa41e747aad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90534941bd5549558e508bdb29ac8265 (127.0.98.2:41491), 41c1bfb87491488db8b4877520e5ac60 (127.0.98.3:38467)
I20260925 04:38:59.506861 2435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcbb16c911b4a30b01af325083e9730" candidate_uuid: "fc188c63638c472e899fcfa41e747aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90534941bd5549558e508bdb29ac8265" is_pre_election: true
I20260925 04:38:59.506949 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcbb16c911b4a30b01af325083e9730" candidate_uuid: "fc188c63638c472e899fcfa41e747aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c1bfb87491488db8b4877520e5ac60" is_pre_election: true
I20260925 04:38:59.507022 2435 raft_consensus.cc:2454] T 5dcbb16c911b4a30b01af325083e9730 P 90534941bd5549558e508bdb29ac8265 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc188c63638c472e899fcfa41e747aad in term 0.
I20260925 04:38:59.507064 2579 raft_consensus.cc:2454] T 5dcbb16c911b4a30b01af325083e9730 P 41c1bfb87491488db8b4877520e5ac60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc188c63638c472e899fcfa41e747aad in term 0.
I20260925 04:38:59.507411 2230 leader_election.cc:302] T 5dcbb16c911b4a30b01af325083e9730 P fc188c63638c472e899fcfa41e747aad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90534941bd5549558e508bdb29ac8265, fc188c63638c472e899fcfa41e747aad; no voters:
I20260925 04:38:59.507695 2731 raft_consensus.cc:2781] T 5dcbb16c911b4a30b01af325083e9730 P fc188c63638c472e899fcfa41e747aad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:38:59.507766 2731 raft_consensus.cc:482] T 5dcbb16c911b4a30b01af325083e9730 P fc188c63638c472e899fcfa41e747aad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:38:59.507803 2731 raft_consensus.cc:3037] T 5dcbb16c911b4a30b01af325083e9730 P fc188c63638c472e899fcfa41e747aad [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:38:59.509097 2731 raft_consensus.cc:504] T 5dcbb16c911b4a30b01af325083e9730 P fc188c63638c472e899fcfa41e747aad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc188c63638c472e899fcfa41e747aad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41049 } } peers { permanent_uuid: "90534941bd5549558e508bdb29ac8265" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "41c1bfb87491488db8b4877520e5ac60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 } }
I20260925 04:38:59.509486 2731 leader_election.cc:288] T 5dcbb16c911b4a30b01af325083e9730 P fc188c63638c472e899fcfa41e747aad [CANDIDATE]: Term 1 election: Requested vote from peers 90534941bd5549558e508bdb29ac8265 (127.0.98.2:41491), 41c1bfb87491488db8b4877520e5ac60 (127.0.98.3:38467)
I20260925 04:38:59.509903 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcbb16c911b4a30b01af325083e9730" candidate_uuid: "fc188c63638c472e899fcfa41e747aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c1bfb87491488db8b4877520e5ac60"
I20260925 04:38:59.509899 2435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcbb16c911b4a30b01af325083e9730" candidate_uuid: "fc188c63638c472e899fcfa41e747aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90534941bd5549558e508bdb29ac8265"
I20260925 04:38:59.510031 2579 raft_consensus.cc:3037] T 5dcbb16c911b4a30b01af325083e9730 P 41c1bfb87491488db8b4877520e5ac60 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:38:59.510031 2435 raft_consensus.cc:3037] T 5dcbb16c911b4a30b01af325083e9730 P 90534941bd5549558e508bdb29ac8265 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:38:59.511168 2435 raft_consensus.cc:2454] T 5dcbb16c911b4a30b01af325083e9730 P 90534941bd5549558e508bdb29ac8265 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc188c63638c472e899fcfa41e747aad in term 1.
I20260925 04:38:59.511168 2579 raft_consensus.cc:2454] T 5dcbb16c911b4a30b01af325083e9730 P 41c1bfb87491488db8b4877520e5ac60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc188c63638c472e899fcfa41e747aad in term 1.
I20260925 04:38:59.511559 2230 leader_election.cc:302] T 5dcbb16c911b4a30b01af325083e9730 P fc188c63638c472e899fcfa41e747aad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41c1bfb87491488db8b4877520e5ac60, fc188c63638c472e899fcfa41e747aad; no voters:
I20260925 04:38:59.511770 2731 raft_consensus.cc:2781] T 5dcbb16c911b4a30b01af325083e9730 P fc188c63638c472e899fcfa41e747aad [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:38:59.511863 2731 raft_consensus.cc:686] T 5dcbb16c911b4a30b01af325083e9730 P fc188c63638c472e899fcfa41e747aad [term 1 LEADER]: Becoming Leader. State: Replica: fc188c63638c472e899fcfa41e747aad, State: Running, Role: LEADER
I20260925 04:38:59.512001 2731 consensus_queue.cc:237] T 5dcbb16c911b4a30b01af325083e9730 P fc188c63638c472e899fcfa41e747aad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc188c63638c472e899fcfa41e747aad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41049 } } peers { permanent_uuid: "90534941bd5549558e508bdb29ac8265" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "41c1bfb87491488db8b4877520e5ac60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 } }
I20260925 04:38:59.513814 2142 catalog_manager.cc:5495] T 5dcbb16c911b4a30b01af325083e9730 P fc188c63638c472e899fcfa41e747aad reported cstate change: term changed from 0 to 1, leader changed from <none> to fc188c63638c472e899fcfa41e747aad (127.0.98.1). New cstate: current_term: 1 leader_uuid: "fc188c63638c472e899fcfa41e747aad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc188c63638c472e899fcfa41e747aad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90534941bd5549558e508bdb29ac8265" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41c1bfb87491488db8b4877520e5ac60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 } health_report { overall_health: UNKNOWN } } }
I20260925 04:38:59.535806 2769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:59.532777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56510 (local address 127.0.98.1:41049)
0925 04:38:59.533034 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:59.533040 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:59.533051 (+ 11us) server_negotiation.cc:407] Connection header received
0925 04:38:59.533087 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:59.533089 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:59.533129 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:59.533199 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:38:59.533448 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:59.533993 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:59.534802 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:59.535018 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:59.535103 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:59.535333 (+ 230us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:38:59.535405 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:38:59.535462 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:59.535556 (+ 94us) server_negotiation.cc:299] Negotiation successful
0925 04:38:59.535642 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":118,"threads_started":1}
I20260925 04:38:59.542208 2579 raft_consensus.cc:1261] T 5dcbb16c911b4a30b01af325083e9730 P 41c1bfb87491488db8b4877520e5ac60 [term 1 FOLLOWER]: Refusing update from remote peer fc188c63638c472e899fcfa41e747aad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:38:59.542939 2731 consensus_queue.cc:1035] T 5dcbb16c911b4a30b01af325083e9730 P fc188c63638c472e899fcfa41e747aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c1bfb87491488db8b4877520e5ac60" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:38:59.543123 2435 raft_consensus.cc:1261] T 5dcbb16c911b4a30b01af325083e9730 P 90534941bd5549558e508bdb29ac8265 [term 1 FOLLOWER]: Refusing update from remote peer fc188c63638c472e899fcfa41e747aad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:38:59.543479 2731 consensus_queue.cc:1035] T 5dcbb16c911b4a30b01af325083e9730 P fc188c63638c472e899fcfa41e747aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "90534941bd5549558e508bdb29ac8265" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:38:59.548276 2721 mvcc.cc:205] Tried to move back new op lower bound from 7333114427561189376 to 7333114427442765824. Current Snapshot: MvccSnapshot[applied={T|T < 7333114427561189376}]
W20260925 04:38:59.557816 2720 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 04:39:01.116074 2140 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:39:01.116228 2140 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:39:01.119041 2140 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3d4badfd4e7f4245a022ad52c0385cce: Sending DeleteTablet for 3 replicas of tablet 5dcbb16c911b4a30b01af325083e9730
I20260925 04:39:01.119812 2279 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dcbb16c911b4a30b01af325083e9730 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:39:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37366
I20260925 04:39:01.119884 2419 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dcbb16c911b4a30b01af325083e9730 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:39:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41962
I20260925 04:39:01.119992 2559 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dcbb16c911b4a30b01af325083e9730 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:39:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36102
I20260925 04:39:01.120198 2795 tablet_replica.cc:330] T 5dcbb16c911b4a30b01af325083e9730 P fc188c63638c472e899fcfa41e747aad: stopping tablet replica
I20260925 04:39:01.120329 2795 raft_consensus.cc:2229] T 5dcbb16c911b4a30b01af325083e9730 P fc188c63638c472e899fcfa41e747aad [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:39:01.120378 2796 tablet_replica.cc:330] T 5dcbb16c911b4a30b01af325083e9730 P 90534941bd5549558e508bdb29ac8265: stopping tablet replica
I20260925 04:39:01.120512 2796 raft_consensus.cc:2229] T 5dcbb16c911b4a30b01af325083e9730 P 90534941bd5549558e508bdb29ac8265 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:39:01.120536 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 2214
I20260925 04:39:01.120548 2795 raft_consensus.cc:2258] T 5dcbb16c911b4a30b01af325083e9730 P fc188c63638c472e899fcfa41e747aad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:39:01.121129 2796 raft_consensus.cc:2258] T 5dcbb16c911b4a30b01af325083e9730 P 90534941bd5549558e508bdb29ac8265 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:39:01.121389 2797 tablet_replica.cc:330] T 5dcbb16c911b4a30b01af325083e9730 P 41c1bfb87491488db8b4877520e5ac60: stopping tablet replica
I20260925 04:39:01.121510 2797 raft_consensus.cc:2229] T 5dcbb16c911b4a30b01af325083e9730 P 41c1bfb87491488db8b4877520e5ac60 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:39:01.121624 2797 raft_consensus.cc:2258] T 5dcbb16c911b4a30b01af325083e9730 P 41c1bfb87491488db8b4877520e5ac60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:39:01.122282 2797 ts_tablet_manager.cc:1905] T 5dcbb16c911b4a30b01af325083e9730 P 41c1bfb87491488db8b4877520e5ac60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:39:01.123276 2796 ts_tablet_manager.cc:1905] T 5dcbb16c911b4a30b01af325083e9730 P 90534941bd5549558e508bdb29ac8265: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:39:01.125209 2797 ts_tablet_manager.cc:1918] T 5dcbb16c911b4a30b01af325083e9730 P 41c1bfb87491488db8b4877520e5ac60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 04:39:01.125308 2797 log.cc:1196] T 5dcbb16c911b4a30b01af325083e9730 P 41c1bfb87491488db8b4877520e5ac60: Deleting WAL directory at /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/ts-2/wal/wals/5dcbb16c911b4a30b01af325083e9730
I20260925 04:39:01.125537 2797 ts_tablet_manager.cc:1939] T 5dcbb16c911b4a30b01af325083e9730 P 41c1bfb87491488db8b4877520e5ac60: Deleting consensus metadata
I20260925 04:39:01.126345 2796 ts_tablet_manager.cc:1918] T 5dcbb16c911b4a30b01af325083e9730 P 90534941bd5549558e508bdb29ac8265: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 04:39:01.126437 2796 log.cc:1196] T 5dcbb16c911b4a30b01af325083e9730 P 90534941bd5549558e508bdb29ac8265: Deleting WAL directory at /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311123478858-392-0/minicluster-data/ts-1/wal/wals/5dcbb16c911b4a30b01af325083e9730
I20260925 04:39:01.126663 2796 ts_tablet_manager.cc:1939] T 5dcbb16c911b4a30b01af325083e9730 P 90534941bd5549558e508bdb29ac8265: Deleting consensus metadata
I20260925 04:39:01.126626 2126 catalog_manager.cc:4841] TS 41c1bfb87491488db8b4877520e5ac60 (127.0.98.3:38467): tablet 5dcbb16c911b4a30b01af325083e9730 (table test-table [id=777feaf30c6240d4aeda0269b8d6a29c]) successfully deleted
W20260925 04:39:01.127751 2128 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41049 (error 108)
I20260925 04:39:01.127898 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 2354
I20260925 04:39:01.130174 2799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:01.129730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41049 (local address 127.0.0.1:56524)
0925 04:39:01.130006 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:01.130080 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":78,"threads_started":1}
W20260925 04:39:01.130326 2128 catalog_manager.cc:4584] TS fc188c63638c472e899fcfa41e747aad (127.0.98.1:41049): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5dcbb16c911b4a30b01af325083e9730: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41049: connect: Connection refused (error 111)
I20260925 04:39:01.134764 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 2494
I20260925 04:39:01.135721 2799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:01.134857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41491 (local address 127.0.0.1:38126)
0925 04:39:01.135530 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:01.135582 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:41491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":595}
I20260925 04:39:01.141310 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 2110
2026-09-25T04:39:01Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6375 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:39:01 dist-test-slave-dc95 krb5kdc[2802](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:39:01 dist-test-slave-dc95 krb5kdc[2802](info): set up 2 sockets
Sep 25 04:39:01 dist-test-slave-dc95 krb5kdc[2802](info): commencing operation
krb5kdc: starting...
W20260925 04:39:02.111459 392 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/krb5kdc/test-user.keytab.
Sep 25 04:39:02 dist-test-slave-dc95 krb5kdc[2802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:39:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:39:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 04:39:02.235927 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32891
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32891
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:39:02.350589 2815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:39:02.350898 2815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:39:02.350939 2815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:39:02.355212 2815 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:39:02.355302 2815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:39:02.355321 2815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:39:02.355338 2815 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:39:02.355352 2815 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:39:02.361111 2815 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:39:02.362227 2815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:39:02.363315 2815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 04:39:02.372870 2815 server_base.cc:1034] running on GCE node
W20260925 04:39:02.372908 2820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:02.372908 2823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:02.372835 2821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:02.373852 2815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:39:02.375109 2815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:39:02.376286 2815 hybrid_clock.cc:648] HybridClock initialized: now 1790311142376271 us; error 34 us; skew 500 ppm
Sep 25 04:39:02 dist-test-slave-dc95 krb5kdc[2802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:39:02.379158 2815 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 04:39:02.380183 2815 webserver.cc:460] Webserver started at http://127.0.98.62:34231/ using document root <none> and password file <none>
I20260925 04:39:02.380743 2815 fs_manager.cc:359] Metadata directory not provided
I20260925 04:39:02.380805 2815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:39:02.380975 2815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:39:02.383116 2815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/minicluster-data/master-0/data/instance:
uuid: "922a52ddfb1a45a1a056b32987ae8fc0"
format_stamp: "Formatted at 2026-09-25 04:39:02 on dist-test-slave-dc95"
server_key: "3a9f0666712c2468a6d87272057439c0"
server_key_iv: "4855a2abed764969adcf1475e86e4d88"
server_key_version: "encryptionkey@0"
I20260925 04:39:02.383716 2815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/minicluster-data/master-0/wal/instance:
uuid: "922a52ddfb1a45a1a056b32987ae8fc0"
format_stamp: "Formatted at 2026-09-25 04:39:02 on dist-test-slave-dc95"
server_key: "3a9f0666712c2468a6d87272057439c0"
server_key_iv: "4855a2abed764969adcf1475e86e4d88"
server_key_version: "encryptionkey@0"
I20260925 04:39:02.387775 2815 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:39:02.390352 2830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:02.391829 2815 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:39:02.391952 2815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/minicluster-data/master-0/wal
uuid: "922a52ddfb1a45a1a056b32987ae8fc0"
format_stamp: "Formatted at 2026-09-25 04:39:02 on dist-test-slave-dc95"
server_key: "3a9f0666712c2468a6d87272057439c0"
server_key_iv: "4855a2abed764969adcf1475e86e4d88"
server_key_version: "encryptionkey@0"
I20260925 04:39:02.392078 2815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:39:02.421554 2815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:39:02.424978 2815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:39:02.425148 2815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:39:02.432066 2815 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32891
I20260925 04:39:02.432173 2882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32891 every 8 connection(s)
I20260925 04:39:02.433148 2815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
I20260925 04:39:02.436189 2883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:39:02.442516 2883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 922a52ddfb1a45a1a056b32987ae8fc0: Bootstrap starting.
I20260925 04:39:02.442946 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 2815
I20260925 04:39:02.443046 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/minicluster-data/master-0/wal/instance
I20260925 04:39:02.443311 392 external_mini_cluster.cc:1428] Setting key 10b52c4c5b060e428cf258582f5e13ea
I20260925 04:39:02.445561 2883 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 922a52ddfb1a45a1a056b32987ae8fc0: Neither blocks nor log segments found. Creating new log.
I20260925 04:39:02.446552 2883 log.cc:824] T 00000000000000000000000000000000 P 922a52ddfb1a45a1a056b32987ae8fc0: Log is configured to *not* fsync() on all Append() calls
I20260925 04:39:02.448932 2883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 922a52ddfb1a45a1a056b32987ae8fc0: No bootstrap required, opened a new log
I20260925 04:39:02.453660 2883 raft_consensus.cc:348] T 00000000000000000000000000000000 P 922a52ddfb1a45a1a056b32987ae8fc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "922a52ddfb1a45a1a056b32987ae8fc0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32891 } }
I20260925 04:39:02.453933 2883 raft_consensus.cc:374] T 00000000000000000000000000000000 P 922a52ddfb1a45a1a056b32987ae8fc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:39:02.453990 2883 raft_consensus.cc:729] T 00000000000000000000000000000000 P 922a52ddfb1a45a1a056b32987ae8fc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 922a52ddfb1a45a1a056b32987ae8fc0, State: Initialized, Role: FOLLOWER
Sep 25 04:39:02 dist-test-slave-dc95 krb5kdc[2802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:39:02.454531 2883 consensus_queue.cc:260] T 00000000000000000000000000000000 P 922a52ddfb1a45a1a056b32987ae8fc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "922a52ddfb1a45a1a056b32987ae8fc0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32891 } }
I20260925 04:39:02.454720 2883 raft_consensus.cc:388] T 00000000000000000000000000000000 P 922a52ddfb1a45a1a056b32987ae8fc0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:39:02.454792 2883 raft_consensus.cc:482] T 00000000000000000000000000000000 P 922a52ddfb1a45a1a056b32987ae8fc0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:39:02.454892 2883 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 922a52ddfb1a45a1a056b32987ae8fc0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:39:02.455873 2883 raft_consensus.cc:504] T 00000000000000000000000000000000 P 922a52ddfb1a45a1a056b32987ae8fc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "922a52ddfb1a45a1a056b32987ae8fc0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32891 } }
I20260925 04:39:02.456223 2883 leader_election.cc:302] T 00000000000000000000000000000000 P 922a52ddfb1a45a1a056b32987ae8fc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 922a52ddfb1a45a1a056b32987ae8fc0; no voters:
I20260925 04:39:02.456542 2883 leader_election.cc:288] T 00000000000000000000000000000000 P 922a52ddfb1a45a1a056b32987ae8fc0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:39:02.456670 2888 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 922a52ddfb1a45a1a056b32987ae8fc0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:39:02.456907 2888 raft_consensus.cc:686] T 00000000000000000000000000000000 P 922a52ddfb1a45a1a056b32987ae8fc0 [term 1 LEADER]: Becoming Leader. State: Replica: 922a52ddfb1a45a1a056b32987ae8fc0, State: Running, Role: LEADER
I20260925 04:39:02.457288 2888 consensus_queue.cc:237] T 00000000000000000000000000000000 P 922a52ddfb1a45a1a056b32987ae8fc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "922a52ddfb1a45a1a056b32987ae8fc0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32891 } }
I20260925 04:39:02.457878 2883 sys_catalog.cc:564] T 00000000000000000000000000000000 P 922a52ddfb1a45a1a056b32987ae8fc0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:39:02.458634 2888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 922a52ddfb1a45a1a056b32987ae8fc0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 922a52ddfb1a45a1a056b32987ae8fc0. Latest consensus state: current_term: 1 leader_uuid: "922a52ddfb1a45a1a056b32987ae8fc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "922a52ddfb1a45a1a056b32987ae8fc0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32891 } } }
I20260925 04:39:02.459285 2889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 922a52ddfb1a45a1a056b32987ae8fc0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "922a52ddfb1a45a1a056b32987ae8fc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "922a52ddfb1a45a1a056b32987ae8fc0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32891 } } }
I20260925 04:39:02.459393 2888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 922a52ddfb1a45a1a056b32987ae8fc0 [sys.catalog]: This master's current role is: LEADER
I20260925 04:39:02.460203 2889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 922a52ddfb1a45a1a056b32987ae8fc0 [sys.catalog]: This master's current role is: LEADER
I20260925 04:39:02.461265 2886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:02.444846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36408 (local address 127.0.98.62:32891)
0925 04:39:02.445336 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:02.445354 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:02.445398 (+ 44us) server_negotiation.cc:407] Connection header received
0925 04:39:02.446336 (+ 938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:02.446362 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:02.446890 (+ 528us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:02.447320 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:02.448191 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:02.449552 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:02.450803 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:02.451387 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:02.451525 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:02.454762 (+ 3237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:02.454769 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:02.454790 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:02.454828 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:02.456637 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:02.457158 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:02.457163 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:02.457169 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:02.457315 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:02.457608 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:02.457611 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:02.457613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:02.458167 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:02.458370 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:02.458699 (+ 329us) server_negotiation.cc:299] Negotiation successful
0925 04:39:02.459045 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":135,"threads_started":1}
W20260925 04:39:02.463133 2902 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 922a52ddfb1a45a1a056b32987ae8fc0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 04:39:02.463197 2902 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 04:39:02.463267 2903 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:39:02.464064 2903 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:39:02.469426 2903 catalog_manager.cc:1347] Generated new cluster ID: 587b8fc8d2d94baca578299c90a570da
I20260925 04:39:02.469506 2903 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:39:02.514784 2903 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:39:02.516312 2903 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:39:02.524472 2903 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 922a52ddfb1a45a1a056b32987ae8fc0: Generated new TSK 0
I20260925 04:39:02.525302 2903 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 04:39:02.563843 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32891
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:39:02.700549 2909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:39:02.700830 2909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:39:02.700874 2909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:39:02.705037 2909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:39:02.705116 2909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:39:02.705185 2909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 04:39:02.710697 2909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:39:02.711854 2909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:39:02.712855 2909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:39:02.722765 2914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:02.722968 2909 server_base.cc:1034] running on GCE node
W20260925 04:39:02.722833 2917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:02.722833 2915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:02.723572 2909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:39:02.724462 2909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:39:02.725651 2909 hybrid_clock.cc:648] HybridClock initialized: now 1790311142725624 us; error 40 us; skew 500 ppm
Sep 25 04:39:02 dist-test-slave-dc95 krb5kdc[2802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:39:02.728653 2909 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 04:39:02.729584 2909 webserver.cc:460] Webserver started at http://127.0.98.1:35893/ using document root <none> and password file <none>
I20260925 04:39:02.730142 2909 fs_manager.cc:359] Metadata directory not provided
I20260925 04:39:02.730204 2909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:39:02.730378 2909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:39:02.732579 2909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-0/data/instance:
uuid: "bbcf8b64338e40619280990c2b9c9595"
format_stamp: "Formatted at 2026-09-25 04:39:02 on dist-test-slave-dc95"
server_key: "d62e1feeedb13e2349e86e828adab9c1"
server_key_iv: "602f84f39b8293b589f63044252ec02d"
server_key_version: "encryptionkey@0"
I20260925 04:39:02.733143 2909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-0/wal/instance:
uuid: "bbcf8b64338e40619280990c2b9c9595"
format_stamp: "Formatted at 2026-09-25 04:39:02 on dist-test-slave-dc95"
server_key: "d62e1feeedb13e2349e86e828adab9c1"
server_key_iv: "602f84f39b8293b589f63044252ec02d"
server_key_version: "encryptionkey@0"
I20260925 04:39:02.737190 2909 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:39:02.739840 2924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:02.741209 2909 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:39:02.741325 2909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-0/wal
uuid: "bbcf8b64338e40619280990c2b9c9595"
format_stamp: "Formatted at 2026-09-25 04:39:02 on dist-test-slave-dc95"
server_key: "d62e1feeedb13e2349e86e828adab9c1"
server_key_iv: "602f84f39b8293b589f63044252ec02d"
server_key_version: "encryptionkey@0"
I20260925 04:39:02.741446 2909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:39:02.756033 2909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:39:02.759395 2909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:39:02.759578 2909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:39:02.760233 2909 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:39:02.761111 2909 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:39:02.761158 2909 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:02.761199 2909 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:39:02.761214 2909 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:02.769954 2909 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36993
I20260925 04:39:02.769972 3037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36993 every 8 connection(s)
I20260925 04:39:02.770943 2909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
I20260925 04:39:02.771332 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 2909
I20260925 04:39:02.771449 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-0/wal/instance
I20260925 04:39:02.771772 392 external_mini_cluster.cc:1428] Setting key fc0435c4c79b140963c244a8a0f093eb
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Sep 25 04:39:02 dist-test-slave-dc95 krb5kdc[2802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:39:02.788326 2886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:02.772973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37281 (local address 127.0.98.62:32891)
0925 04:39:02.773160 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:02.773165 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:02.774285 (+ 1120us) server_negotiation.cc:407] Connection header received
0925 04:39:02.775791 (+ 1506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:02.775795 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:02.775855 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:02.775932 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:02.777295 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:02.777825 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:02.779042 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:02.779280 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:02.779375 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:02.784482 (+ 5107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:02.784485 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:02.784487 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:02.784517 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:02.786031 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:02.786635 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:02.786638 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:02.786639 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:02.786746 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:02.787129 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:02.787133 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:02.787135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:02.787334 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:02.787422 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:02.788018 (+ 596us) server_negotiation.cc:299] Negotiation successful
0925 04:39:02.788157 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 04:39:02.789299 3040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:02.773503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32891 (local address 127.0.98.1:37281)
0925 04:39:02.774119 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:02.774167 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:02.775523 (+ 1356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:02.776149 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:02.776157 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:02.776542 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:02.777137 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:02.777155 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:02.777956 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:02.777960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:02.778909 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:02.778919 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:02.779445 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:02.779450 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:02.779664 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:02.780261 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:02.780278 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:02.784236 (+ 3958us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:39:02.786199 (+ 1963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:02.786205 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:02.786222 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:02.786520 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:02.786843 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:02.786847 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:02.786850 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:02.787022 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:02.787442 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:39:02.787449 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:39:02.787773 (+ 324us) client_negotiation.cc:769] Sending connection context
0925 04:39:02.787962 (+ 189us) client_negotiation.cc:240] Negotiation successful
0925 04:39:02.788345 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":165,"threads_started":1}
I20260925 04:39:02.790644 3038 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32891
I20260925 04:39:02.790905 3038 heartbeater.cc:464] Registering TS with master...
I20260925 04:39:02.791456 3038 heartbeater.cc:511] Master 127.0.98.62:32891 requested a full tablet report, sending...
I20260925 04:39:02.793138 2847 ts_manager.cc:194] Registered new tserver with Master: bbcf8b64338e40619280990c2b9c9595 (127.0.98.1:36993)
I20260925 04:39:02.794358 2847 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37281
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 04:39:02.799726 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32891
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:39:02.913272 3043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:39:02.913621 3043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:39:02.913689 3043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:39:02.918200 3043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:39:02.918306 3043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:39:02.918394 3043 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 04:39:02.924085 3043 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3043
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:39:02.925223 3043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:39:02.926272 3043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 04:39:02.936309 3043 server_base.cc:1034] running on GCE node
W20260925 04:39:02.936249 3048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:02.936264 3049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:02.936249 3051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:02.936972 3043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:39:02.937620 3043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:39:02.938804 3043 hybrid_clock.cc:648] HybridClock initialized: now 1790311142938782 us; error 35 us; skew 500 ppm
Sep 25 04:39:02 dist-test-slave-dc95 krb5kdc[2802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:39:02.941704 3043 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 04:39:02.942670 3043 webserver.cc:460] Webserver started at http://127.0.98.2:38335/ using document root <none> and password file <none>
I20260925 04:39:02.943239 3043 fs_manager.cc:359] Metadata directory not provided
I20260925 04:39:02.943310 3043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:39:02.943536 3043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:39:02.945955 3043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-1/data/instance:
uuid: "6391ee1de02b43da9ccf8fe7d86d9ef9"
format_stamp: "Formatted at 2026-09-25 04:39:02 on dist-test-slave-dc95"
server_key: "7214b195bfa408cf91bc522fcfc2c720"
server_key_iv: "516c40f6ae53a93accebe6ac9b0e0283"
server_key_version: "encryptionkey@0"
I20260925 04:39:02.946556 3043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6391ee1de02b43da9ccf8fe7d86d9ef9"
format_stamp: "Formatted at 2026-09-25 04:39:02 on dist-test-slave-dc95"
server_key: "7214b195bfa408cf91bc522fcfc2c720"
server_key_iv: "516c40f6ae53a93accebe6ac9b0e0283"
server_key_version: "encryptionkey@0"
I20260925 04:39:02.950703 3043 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260925 04:39:02.953469 3058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:02.954862 3043 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:39:02.954998 3043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-1/wal
uuid: "6391ee1de02b43da9ccf8fe7d86d9ef9"
format_stamp: "Formatted at 2026-09-25 04:39:02 on dist-test-slave-dc95"
server_key: "7214b195bfa408cf91bc522fcfc2c720"
server_key_iv: "516c40f6ae53a93accebe6ac9b0e0283"
server_key_version: "encryptionkey@0"
I20260925 04:39:02.955114 3043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:39:02.966845 3043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:39:02.970531 3043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:39:02.970752 3043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:39:02.971414 3043 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:39:02.972467 3043 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:39:02.972522 3043 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:02.972581 3043 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:39:02.972611 3043 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:02.981690 3043 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35239
I20260925 04:39:02.981721 3171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35239 every 8 connection(s)
I20260925 04:39:02.982625 3043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
I20260925 04:39:02.986287 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 3043
I20260925 04:39:02.986415 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-1/wal/instance
I20260925 04:39:02.986701 392 external_mini_cluster.cc:1428] Setting key 583e9bbf958e22e5bb967805e5e8ed0a
Sep 25 04:39:02 dist-test-slave-dc95 krb5kdc[2802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260925 04:39:02.998162 2886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:02.984665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43327 (local address 127.0.98.62:32891)
0925 04:39:02.984813 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:02.984817 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:02.985705 (+ 888us) server_negotiation.cc:407] Connection header received
0925 04:39:02.987439 (+ 1734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:02.987442 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:02.987504 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:02.987579 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:02.989177 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:02.989725 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:02.990978 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:02.991310 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:02.991416 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:02.994276 (+ 2860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:02.994280 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:02.994282 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:02.994310 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:02.995798 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:02.996475 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:02.996478 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:02.996479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:02.996575 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:02.997025 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:02.997028 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:02.997030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:02.997245 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:02.997347 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:02.997913 (+ 566us) server_negotiation.cc:299] Negotiation successful
0925 04:39:02.998032 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260925 04:39:02.999260 3174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:02.985025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32891 (local address 127.0.98.2:43327)
0925 04:39:02.985528 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:02.985576 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:02.987193 (+ 1617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:02.987826 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:02.987834 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:02.988241 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:02.988953 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:02.988975 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:02.989904 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:02.989908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:02.990845 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:02.990856 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:02.991475 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:02.991478 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:02.991685 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:02.992297 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:02.992314 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:02.993963 (+ 1649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:39:02.995968 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:02.995974 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:02.995989 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:02.996350 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:02.996700 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:02.996703 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:02.996705 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:02.996882 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:02.997369 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:39:02.997376 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:39:02.997665 (+ 289us) client_negotiation.cc:769] Sending connection context
0925 04:39:02.997847 (+ 182us) client_negotiation.cc:240] Negotiation successful
0925 04:39:02.998182 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":127,"threads_started":1}
I20260925 04:39:03.000494 3172 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32891
I20260925 04:39:03.000830 3172 heartbeater.cc:464] Registering TS with master...
I20260925 04:39:03.001506 3172 heartbeater.cc:511] Master 127.0.98.62:32891 requested a full tablet report, sending...
I20260925 04:39:03.002724 2847 ts_manager.cc:194] Registered new tserver with Master: 6391ee1de02b43da9ccf8fe7d86d9ef9 (127.0.98.2:35239)
I20260925 04:39:03.003373 2847 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43327
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 04:39:03.017102 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32891
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:39:03.133934 3177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:39:03.134222 3177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:39:03.134294 3177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:39:03.138664 3177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:39:03.138761 3177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:39:03.138844 3177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 04:39:03.144681 3177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:39:03.145880 3177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:39:03.146984 3177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:39:03.157364 3183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:03.157413 3182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:03.157603 3177 server_base.cc:1034] running on GCE node
W20260925 04:39:03.157739 3185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:03.158156 3177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:39:03.158910 3177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:39:03.160074 3177 hybrid_clock.cc:648] HybridClock initialized: now 1790311143160056 us; error 34 us; skew 500 ppm
Sep 25 04:39:03 dist-test-slave-dc95 krb5kdc[2802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:39:03.163201 3177 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 04:39:03.164216 3177 webserver.cc:460] Webserver started at http://127.0.98.3:33919/ using document root <none> and password file <none>
I20260925 04:39:03.164788 3177 fs_manager.cc:359] Metadata directory not provided
I20260925 04:39:03.164858 3177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:39:03.165037 3177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:39:03.167193 3177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-2/data/instance:
uuid: "e05d38f9a4f44696ad8d134db13b57da"
format_stamp: "Formatted at 2026-09-25 04:39:03 on dist-test-slave-dc95"
server_key: "5ae253b8879593821e18df1f89d0e0f4"
server_key_iv: "8159e9f90d78cd150692adeb1be08f7f"
server_key_version: "encryptionkey@0"
I20260925 04:39:03.167814 3177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e05d38f9a4f44696ad8d134db13b57da"
format_stamp: "Formatted at 2026-09-25 04:39:03 on dist-test-slave-dc95"
server_key: "5ae253b8879593821e18df1f89d0e0f4"
server_key_iv: "8159e9f90d78cd150692adeb1be08f7f"
server_key_version: "encryptionkey@0"
I20260925 04:39:03.171947 3177 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:39:03.174702 3192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:03.176223 3177 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:39:03.176352 3177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-2/wal
uuid: "e05d38f9a4f44696ad8d134db13b57da"
format_stamp: "Formatted at 2026-09-25 04:39:03 on dist-test-slave-dc95"
server_key: "5ae253b8879593821e18df1f89d0e0f4"
server_key_iv: "8159e9f90d78cd150692adeb1be08f7f"
server_key_version: "encryptionkey@0"
I20260925 04:39:03.176483 3177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:39:03.189869 3177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:39:03.193279 3177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:39:03.193481 3177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:39:03.194085 3177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:39:03.194941 3177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:39:03.194990 3177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:03.195057 3177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:39:03.195084 3177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:03.203807 3177 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35965
I20260925 04:39:03.203833 3305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35965 every 8 connection(s)
I20260925 04:39:03.204736 3177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
I20260925 04:39:03.214000 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 3177
I20260925 04:39:03.214111 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-2/wal/instance
I20260925 04:39:03.214375 392 external_mini_cluster.cc:1428] Setting key 70c87992adbfb9a83432f535a3facade
Sep 25 04:39:03 dist-test-slave-dc95 krb5kdc[2802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:39:03.218744 2886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:03.206634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37943 (local address 127.0.98.62:32891)
0925 04:39:03.206769 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:03.206773 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:03.207775 (+ 1002us) server_negotiation.cc:407] Connection header received
0925 04:39:03.209196 (+ 1421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:03.209200 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:03.209261 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:03.209333 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:03.210569 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:03.211116 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:03.212353 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:03.212575 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:03.212667 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:03.215231 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:03.215235 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:03.215238 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:03.215270 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:03.216790 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:03.217315 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:03.217318 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:03.217319 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:03.217416 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:03.217703 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:03.217705 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:03.217707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:03.217905 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:03.217979 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:03.218489 (+ 510us) server_negotiation.cc:299] Negotiation successful
0925 04:39:03.218613 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 04:39:03.219556 3308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:03.207030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32891 (local address 127.0.98.3:37943)
0925 04:39:03.207582 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:03.207666 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:03.208965 (+ 1299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:03.209523 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:03.209530 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:03.209909 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:03.210415 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:03.210429 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:03.211240 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:03.211243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:03.212216 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:03.212227 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:03.212733 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:03.212738 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:03.212907 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:03.213509 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:03.213525 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:03.214999 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:39:03.216899 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:03.216904 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:03.216918 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:03.217185 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:03.217493 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:03.217495 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:03.217497 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:03.217631 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:03.217978 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:39:03.217983 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:39:03.218229 (+ 246us) client_negotiation.cc:769] Sending connection context
0925 04:39:03.218409 (+ 180us) client_negotiation.cc:240] Negotiation successful
0925 04:39:03.218679 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":121,"threads_started":1}
I20260925 04:39:03.220731 3306 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32891
I20260925 04:39:03.221005 3306 heartbeater.cc:464] Registering TS with master...
I20260925 04:39:03.221517 3306 heartbeater.cc:511] Master 127.0.98.62:32891 requested a full tablet report, sending...
I20260925 04:39:03.222613 2847 ts_manager.cc:194] Registered new tserver with Master: e05d38f9a4f44696ad8d134db13b57da (127.0.98.3:35965)
I20260925 04:39:03.223193 2847 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37943
I20260925 04:39:03.228426 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 04:39:03.240813 2886 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:39:03.239249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36420 (local address 127.0.98.62:32891)
0925 04:39:03.239386 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:03.239390 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:03.239567 (+ 177us) server_negotiation.cc:407] Connection header received
0925 04:39:03.239719 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:03.239721 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:03.239768 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:03.239849 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:03.240691 (+ 842us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36420: BlockingRecv error: recv got EOF from 127.0.0.1:36420 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 04:39:03.242044 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 2909
I20260925 04:39:03.247061 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 3043
I20260925 04:39:03.251816 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 3177
I20260925 04:39:03.256211 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 2815
2026-09-25T04:39:03Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2113 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790311123478858-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:39:03 dist-test-slave-dc95 krb5kdc[3318](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:39:03 dist-test-slave-dc95 krb5kdc[3318](info): set up 2 sockets
Sep 25 04:39:03 dist-test-slave-dc95 krb5kdc[3318](info): commencing operation
krb5kdc: starting...
W20260925 04:39:04.223444 392 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790311123478858-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790311123478858-392-0/krb5kdc/test-user.keytab.
Sep 25 04:39:04 dist-test-slave-dc95 krb5kdc[3318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:39:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:39:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 04:39:04.346861 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790311123478858-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790311123478858-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790311123478858-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35397
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35397
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:39:04.461630 3331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:39:04.461915 3331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:39:04.461956 3331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:39:04.466179 3331 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:39:04.466257 3331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:39:04.466277 3331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:39:04.466293 3331 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:39:04.466307 3331 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:39:04.471864 3331 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790311123478858-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790311123478858-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35397
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790311123478858-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:39:04.473019 3331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:39:04.474154 3331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:39:04.483207 3336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:04.483227 3339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:04.483212 3337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:04.483631 3331 server_base.cc:1034] running on GCE node
I20260925 04:39:04.484284 3331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:39:04.485532 3331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:39:04.486718 3331 hybrid_clock.cc:648] HybridClock initialized: now 1790311144486696 us; error 67 us; skew 500 ppm
Sep 25 04:39:04 dist-test-slave-dc95 krb5kdc[3318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:39:04.489738 3331 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 04:39:04.490769 3331 webserver.cc:460] Webserver started at http://127.0.98.62:38641/ using document root <none> and password file <none>
I20260925 04:39:04.491369 3331 fs_manager.cc:359] Metadata directory not provided
I20260925 04:39:04.491443 3331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:39:04.491688 3331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:39:04.493832 3331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790311123478858-392-0/minicluster-data/master-0/data/instance:
uuid: "006fad0fdd6a4a5289fa97d2eeb5c4af"
format_stamp: "Formatted at 2026-09-25 04:39:04 on dist-test-slave-dc95"
server_key: "89a81dee9d0cd79275f7fb3aa1617ffc"
server_key_iv: "41266fe2dd3fd22a56e8e69c313ca399"
server_key_version: "encryptionkey@0"
I20260925 04:39:04.494431 3331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790311123478858-392-0/minicluster-data/master-0/wal/instance:
uuid: "006fad0fdd6a4a5289fa97d2eeb5c4af"
format_stamp: "Formatted at 2026-09-25 04:39:04 on dist-test-slave-dc95"
server_key: "89a81dee9d0cd79275f7fb3aa1617ffc"
server_key_iv: "41266fe2dd3fd22a56e8e69c313ca399"
server_key_version: "encryptionkey@0"
I20260925 04:39:04.498346 3331 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260925 04:39:04.501104 3346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:04.502585 3331 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 04:39:04.502720 3331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790311123478858-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790311123478858-392-0/minicluster-data/master-0/wal
uuid: "006fad0fdd6a4a5289fa97d2eeb5c4af"
format_stamp: "Formatted at 2026-09-25 04:39:04 on dist-test-slave-dc95"
server_key: "89a81dee9d0cd79275f7fb3aa1617ffc"
server_key_iv: "41266fe2dd3fd22a56e8e69c313ca399"
server_key_version: "encryptionkey@0"
I20260925 04:39:04.502864 3331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790311123478858-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790311123478858-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790311123478858-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:39:04.520437 3331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:39:04.524134 3331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:39:04.524339 3331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:39:04.531323 3331 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35397
I20260925 04:39:04.531587 3398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35397 every 8 connection(s)
I20260925 04:39:04.532351 3331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
I20260925 04:39:04.534778 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 3331
I20260925 04:39:04.534893 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790311123478858-392-0/minicluster-data/master-0/wal/instance
I20260925 04:39:04.535137 392 external_mini_cluster.cc:1428] Setting key a38237c4b726fdb85fddd1108b4b55d6
I20260925 04:39:04.535306 3399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:39:04.542742 3399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 006fad0fdd6a4a5289fa97d2eeb5c4af: Bootstrap starting.
Sep 25 04:39:04 dist-test-slave-dc95 krb5kdc[3318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:39:04.545442 3399 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 006fad0fdd6a4a5289fa97d2eeb5c4af: Neither blocks nor log segments found. Creating new log.
I20260925 04:39:04.546361 3399 log.cc:824] T 00000000000000000000000000000000 P 006fad0fdd6a4a5289fa97d2eeb5c4af: Log is configured to *not* fsync() on all Append() calls
I20260925 04:39:04.548736 3399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 006fad0fdd6a4a5289fa97d2eeb5c4af: No bootstrap required, opened a new log
I20260925 04:39:04.550189 3402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:04.536528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51406 (local address 127.0.98.62:35397)
0925 04:39:04.536978 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:04.536998 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:04.537041 (+ 43us) server_negotiation.cc:407] Connection header received
0925 04:39:04.537750 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:04.537776 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:04.538244 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:04.538650 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:04.539303 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:04.540787 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:04.541686 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:04.542368 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:04.542521 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:04.545140 (+ 2619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:04.545146 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:04.545163 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:04.545198 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:04.547010 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:04.547433 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:04.547470 (+ 37us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:04.547476 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:04.547663 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:04.547920 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:04.547923 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:04.547925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:04.548436 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:04.548621 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:04.548887 (+ 266us) server_negotiation.cc:299] Negotiation successful
0925 04:39:04.549193 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":118,"threads_started":1}
I20260925 04:39:04.553184 3399 raft_consensus.cc:348] T 00000000000000000000000000000000 P 006fad0fdd6a4a5289fa97d2eeb5c4af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "006fad0fdd6a4a5289fa97d2eeb5c4af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35397 } }
I20260925 04:39:04.553429 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:39:04.553459 3399 raft_consensus.cc:374] T 00000000000000000000000000000000 P 006fad0fdd6a4a5289fa97d2eeb5c4af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:39:04.553515 3399 raft_consensus.cc:729] T 00000000000000000000000000000000 P 006fad0fdd6a4a5289fa97d2eeb5c4af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 006fad0fdd6a4a5289fa97d2eeb5c4af, State: Initialized, Role: FOLLOWER
I20260925 04:39:04.553933 3399 consensus_queue.cc:260] T 00000000000000000000000000000000 P 006fad0fdd6a4a5289fa97d2eeb5c4af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "006fad0fdd6a4a5289fa97d2eeb5c4af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35397 } }
I20260925 04:39:04.554049 3399 raft_consensus.cc:388] T 00000000000000000000000000000000 P 006fad0fdd6a4a5289fa97d2eeb5c4af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:39:04.554111 3399 raft_consensus.cc:482] T 00000000000000000000000000000000 P 006fad0fdd6a4a5289fa97d2eeb5c4af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:39:04.554201 3399 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 006fad0fdd6a4a5289fa97d2eeb5c4af [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:39:04.555157 3399 raft_consensus.cc:504] T 00000000000000000000000000000000 P 006fad0fdd6a4a5289fa97d2eeb5c4af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "006fad0fdd6a4a5289fa97d2eeb5c4af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35397 } }
I20260925 04:39:04.555557 3399 leader_election.cc:302] T 00000000000000000000000000000000 P 006fad0fdd6a4a5289fa97d2eeb5c4af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 006fad0fdd6a4a5289fa97d2eeb5c4af; no voters:
I20260925 04:39:04.555938 3399 leader_election.cc:288] T 00000000000000000000000000000000 P 006fad0fdd6a4a5289fa97d2eeb5c4af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:39:04.556020 3405 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 006fad0fdd6a4a5289fa97d2eeb5c4af [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:39:04.556238 3405 raft_consensus.cc:686] T 00000000000000000000000000000000 P 006fad0fdd6a4a5289fa97d2eeb5c4af [term 1 LEADER]: Becoming Leader. State: Replica: 006fad0fdd6a4a5289fa97d2eeb5c4af, State: Running, Role: LEADER
I20260925 04:39:04.556556 3405 consensus_queue.cc:237] T 00000000000000000000000000000000 P 006fad0fdd6a4a5289fa97d2eeb5c4af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "006fad0fdd6a4a5289fa97d2eeb5c4af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35397 } }
I20260925 04:39:04.557190 3399 sys_catalog.cc:564] T 00000000000000000000000000000000 P 006fad0fdd6a4a5289fa97d2eeb5c4af [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:39:04.558071 3406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 006fad0fdd6a4a5289fa97d2eeb5c4af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "006fad0fdd6a4a5289fa97d2eeb5c4af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "006fad0fdd6a4a5289fa97d2eeb5c4af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35397 } } }
I20260925 04:39:04.558192 3406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 006fad0fdd6a4a5289fa97d2eeb5c4af [sys.catalog]: This master's current role is: LEADER
I20260925 04:39:04.558557 3407 sys_catalog.cc:455] T 00000000000000000000000000000000 P 006fad0fdd6a4a5289fa97d2eeb5c4af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 006fad0fdd6a4a5289fa97d2eeb5c4af. Latest consensus state: current_term: 1 leader_uuid: "006fad0fdd6a4a5289fa97d2eeb5c4af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "006fad0fdd6a4a5289fa97d2eeb5c4af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35397 } } }
I20260925 04:39:04.558691 3407 sys_catalog.cc:458] T 00000000000000000000000000000000 P 006fad0fdd6a4a5289fa97d2eeb5c4af [sys.catalog]: This master's current role is: LEADER
I20260925 04:39:04.558987 3414 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:39:04.562767 3414 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:39:04.569010 3414 catalog_manager.cc:1347] Generated new cluster ID: db1d8f1058d7483fbc4e6ff66ddc218f
I20260925 04:39:04.569088 3414 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:39:04.573728 3402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:04.568961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51412 (local address 127.0.98.62:35397)
0925 04:39:04.569083 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:04.569087 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:04.569382 (+ 295us) server_negotiation.cc:407] Connection header received
0925 04:39:04.569525 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:04.569528 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:04.569576 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:04.569626 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:04.570258 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:04.570786 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:04.571677 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:04.571949 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:04.572038 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:04.572828 (+ 790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:04.572832 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:04.572835 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:39:04.573285 (+ 450us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:04.573315 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:39:04.573331 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:04.573409 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:04.573511 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 04:39:04.573576 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":49}
I20260925 04:39:04.604444 3414 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:39:04.605444 3414 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:39:04.611953 3414 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 006fad0fdd6a4a5289fa97d2eeb5c4af: Generated new TSK 0
I20260925 04:39:04.612663 3414 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:39:04.622323 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 3331
2026-09-25T04:39:04Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1364 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:39:04 dist-test-slave-dc95 krb5kdc[3432](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:39:04 dist-test-slave-dc95 krb5kdc[3432](info): set up 2 sockets
Sep 25 04:39:04 dist-test-slave-dc95 krb5kdc[3432](info): commencing operation
krb5kdc: starting...
W20260925 04:39:05.587594 392 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/krb5kdc/test-user.keytab.
Sep 25 04:39:05 dist-test-slave-dc95 krb5kdc[3432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:39:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:39:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 04:39:05.711184 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39081
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39081
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:39:05.826318 3445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:39:05.826647 3445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:39:05.826705 3445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:39:05.831001 3445 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:39:05.831095 3445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:39:05.831122 3445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:39:05.831143 3445 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:39:05.831163 3445 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:39:05.836833 3445 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39081
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39081
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:39:05.837977 3445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:39:05.839046 3445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:39:05.848089 3450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:05.848089 3451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:05.848111 3453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:05.848536 3445 server_base.cc:1034] running on GCE node
I20260925 04:39:05.849460 3445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:39:05.850740 3445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:39:05.851926 3445 hybrid_clock.cc:648] HybridClock initialized: now 1790311145851908 us; error 38 us; skew 500 ppm
Sep 25 04:39:05 dist-test-slave-dc95 krb5kdc[3432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311145, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:39:05.855141 3445 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 04:39:05.856252 3445 webserver.cc:460] Webserver started at http://127.0.98.62:43509/ using document root <none> and password file <none>
I20260925 04:39:05.856896 3445 fs_manager.cc:359] Metadata directory not provided
I20260925 04:39:05.856976 3445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:39:05.857211 3445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:39:05.859442 3445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/minicluster-data/master-0/data/instance:
uuid: "efef804ce051421f9f2c65eb3167f7a6"
format_stamp: "Formatted at 2026-09-25 04:39:05 on dist-test-slave-dc95"
server_key: "5f8e6c22b3578282249ad1d88a5117f4"
server_key_iv: "54882e2f05c6a8189722d816b35652b7"
server_key_version: "encryptionkey@0"
I20260925 04:39:05.860086 3445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/minicluster-data/master-0/wal/instance:
uuid: "efef804ce051421f9f2c65eb3167f7a6"
format_stamp: "Formatted at 2026-09-25 04:39:05 on dist-test-slave-dc95"
server_key: "5f8e6c22b3578282249ad1d88a5117f4"
server_key_iv: "54882e2f05c6a8189722d816b35652b7"
server_key_version: "encryptionkey@0"
I20260925 04:39:05.870919 3445 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.004s sys 0.000s
I20260925 04:39:05.873970 3460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:05.875520 3445 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 04:39:05.875720 3445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/minicluster-data/master-0/wal
uuid: "efef804ce051421f9f2c65eb3167f7a6"
format_stamp: "Formatted at 2026-09-25 04:39:05 on dist-test-slave-dc95"
server_key: "5f8e6c22b3578282249ad1d88a5117f4"
server_key_iv: "54882e2f05c6a8189722d816b35652b7"
server_key_version: "encryptionkey@0"
I20260925 04:39:05.875845 3445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:39:05.885466 3445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:39:05.889343 3445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:39:05.889552 3445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:39:05.897392 3445 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39081
I20260925 04:39:05.897667 3512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39081 every 8 connection(s)
I20260925 04:39:05.898540 3445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
I20260925 04:39:05.901669 3513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:39:05.907599 3513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efef804ce051421f9f2c65eb3167f7a6: Bootstrap starting.
I20260925 04:39:05.908524 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 3445
I20260925 04:39:05.908643 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/minicluster-data/master-0/wal/instance
I20260925 04:39:05.908885 392 external_mini_cluster.cc:1428] Setting key 75a44608997da8a80eb0fbf2a07b3dde
I20260925 04:39:05.910579 3513 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P efef804ce051421f9f2c65eb3167f7a6: Neither blocks nor log segments found. Creating new log.
I20260925 04:39:05.911564 3513 log.cc:824] T 00000000000000000000000000000000 P efef804ce051421f9f2c65eb3167f7a6: Log is configured to *not* fsync() on all Append() calls
I20260925 04:39:05.914057 3513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efef804ce051421f9f2c65eb3167f7a6: No bootstrap required, opened a new log
I20260925 04:39:05.918258 3513 raft_consensus.cc:348] T 00000000000000000000000000000000 P efef804ce051421f9f2c65eb3167f7a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efef804ce051421f9f2c65eb3167f7a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39081 } }
I20260925 04:39:05.918601 3513 raft_consensus.cc:374] T 00000000000000000000000000000000 P efef804ce051421f9f2c65eb3167f7a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:39:05.918674 3513 raft_consensus.cc:729] T 00000000000000000000000000000000 P efef804ce051421f9f2c65eb3167f7a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efef804ce051421f9f2c65eb3167f7a6, State: Initialized, Role: FOLLOWER
Sep 25 04:39:05 dist-test-slave-dc95 krb5kdc[3432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:39:05.919153 3513 consensus_queue.cc:260] T 00000000000000000000000000000000 P efef804ce051421f9f2c65eb3167f7a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efef804ce051421f9f2c65eb3167f7a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39081 } }
I20260925 04:39:05.919302 3513 raft_consensus.cc:388] T 00000000000000000000000000000000 P efef804ce051421f9f2c65eb3167f7a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:39:05.919373 3513 raft_consensus.cc:482] T 00000000000000000000000000000000 P efef804ce051421f9f2c65eb3167f7a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:39:05.919474 3513 raft_consensus.cc:3037] T 00000000000000000000000000000000 P efef804ce051421f9f2c65eb3167f7a6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:39:05.920770 3513 raft_consensus.cc:504] T 00000000000000000000000000000000 P efef804ce051421f9f2c65eb3167f7a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efef804ce051421f9f2c65eb3167f7a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39081 } }
I20260925 04:39:05.921185 3513 leader_election.cc:302] T 00000000000000000000000000000000 P efef804ce051421f9f2c65eb3167f7a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: efef804ce051421f9f2c65eb3167f7a6; no voters:
I20260925 04:39:05.921592 3513 leader_election.cc:288] T 00000000000000000000000000000000 P efef804ce051421f9f2c65eb3167f7a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:39:05.921705 3518 raft_consensus.cc:2781] T 00000000000000000000000000000000 P efef804ce051421f9f2c65eb3167f7a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:39:05.921967 3518 raft_consensus.cc:686] T 00000000000000000000000000000000 P efef804ce051421f9f2c65eb3167f7a6 [term 1 LEADER]: Becoming Leader. State: Replica: efef804ce051421f9f2c65eb3167f7a6, State: Running, Role: LEADER
I20260925 04:39:05.922401 3518 consensus_queue.cc:237] T 00000000000000000000000000000000 P efef804ce051421f9f2c65eb3167f7a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efef804ce051421f9f2c65eb3167f7a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39081 } }
I20260925 04:39:05.923059 3513 sys_catalog.cc:564] T 00000000000000000000000000000000 P efef804ce051421f9f2c65eb3167f7a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:39:05.924484 3520 sys_catalog.cc:455] T 00000000000000000000000000000000 P efef804ce051421f9f2c65eb3167f7a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader efef804ce051421f9f2c65eb3167f7a6. Latest consensus state: current_term: 1 leader_uuid: "efef804ce051421f9f2c65eb3167f7a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efef804ce051421f9f2c65eb3167f7a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39081 } } }
I20260925 04:39:05.924697 3520 sys_catalog.cc:458] T 00000000000000000000000000000000 P efef804ce051421f9f2c65eb3167f7a6 [sys.catalog]: This master's current role is: LEADER
I20260925 04:39:05.925041 3519 sys_catalog.cc:455] T 00000000000000000000000000000000 P efef804ce051421f9f2c65eb3167f7a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "efef804ce051421f9f2c65eb3167f7a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efef804ce051421f9f2c65eb3167f7a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39081 } } }
I20260925 04:39:05.925230 3519 sys_catalog.cc:458] T 00000000000000000000000000000000 P efef804ce051421f9f2c65eb3167f7a6 [sys.catalog]: This master's current role is: LEADER
I20260925 04:39:05.925689 3516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:05.910116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59112 (local address 127.0.98.62:39081)
0925 04:39:05.910629 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:05.910676 (+ 47us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:05.910723 (+ 47us) server_negotiation.cc:407] Connection header received
0925 04:39:05.911713 (+ 990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:05.911740 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:05.912208 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:05.912583 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:05.913323 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:05.914745 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:05.915707 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:05.916435 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:05.916604 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:05.919373 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:05.919380 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:05.919401 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:05.919443 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:05.921623 (+ 2180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:05.922276 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:05.922282 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:05.922288 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:05.922438 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:05.922762 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:05.922765 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:05.922767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:05.923373 (+ 606us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:05.923591 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:05.924317 (+ 726us) server_negotiation.cc:299] Negotiation successful
0925 04:39:05.924659 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":142,"threads_started":1}
I20260925 04:39:05.925942 3530 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:39:05.929293 3530 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:39:05.935422 3530 catalog_manager.cc:1347] Generated new cluster ID: eedb918062fa47ff94fdf9707345449e
I20260925 04:39:05.935501 3530 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:39:05.941346 3530 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:39:05.942461 3530 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:39:05.952626 3530 catalog_manager.cc:5868] T 00000000000000000000000000000000 P efef804ce051421f9f2c65eb3167f7a6: Generated new TSK 0
I20260925 04:39:05.953415 3530 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 04:39:05.992347 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39081
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:39:06.134073 3539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:39:06.134397 3539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:39:06.134441 3539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:39:06.138731 3539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:39:06.138816 3539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:39:06.138877 3539 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 04:39:06.144629 3539 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:39:06.145835 3539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:39:06.146907 3539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:39:06.157464 3545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:06.157542 3539 server_base.cc:1034] running on GCE node
W20260925 04:39:06.157472 3547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:06.157495 3544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:06.158277 3539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:39:06.159027 3539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:39:06.160231 3539 hybrid_clock.cc:648] HybridClock initialized: now 1790311146160218 us; error 34 us; skew 500 ppm
Sep 25 04:39:06 dist-test-slave-dc95 krb5kdc[3432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311146, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:39:06.163708 3539 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 04:39:06.164768 3539 webserver.cc:460] Webserver started at http://127.0.98.1:40183/ using document root <none> and password file <none>
I20260925 04:39:06.165403 3539 fs_manager.cc:359] Metadata directory not provided
I20260925 04:39:06.165483 3539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:39:06.165719 3539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:39:06.168044 3539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-0/data/instance:
uuid: "9a9ce405fe5040a0bef7608fe3f23379"
format_stamp: "Formatted at 2026-09-25 04:39:06 on dist-test-slave-dc95"
server_key: "2ecaa6bbbf726d29aa47cc0d932deed5"
server_key_iv: "0620cb63723b6ee61d02fd312e1033b7"
server_key_version: "encryptionkey@0"
I20260925 04:39:06.168689 3539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9a9ce405fe5040a0bef7608fe3f23379"
format_stamp: "Formatted at 2026-09-25 04:39:06 on dist-test-slave-dc95"
server_key: "2ecaa6bbbf726d29aa47cc0d932deed5"
server_key_iv: "0620cb63723b6ee61d02fd312e1033b7"
server_key_version: "encryptionkey@0"
I20260925 04:39:06.172992 3539 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:39:06.176246 3554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:06.177728 3539 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:39:06.177872 3539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-0/wal
uuid: "9a9ce405fe5040a0bef7608fe3f23379"
format_stamp: "Formatted at 2026-09-25 04:39:06 on dist-test-slave-dc95"
server_key: "2ecaa6bbbf726d29aa47cc0d932deed5"
server_key_iv: "0620cb63723b6ee61d02fd312e1033b7"
server_key_version: "encryptionkey@0"
I20260925 04:39:06.177999 3539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:39:06.214584 3539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:39:06.218170 3539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:39:06.218389 3539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:39:06.219019 3539 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:39:06.219952 3539 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:39:06.220006 3539 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:06.220062 3539 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:39:06.220091 3539 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:06.230263 3539 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34865
I20260925 04:39:06.230507 3667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34865 every 8 connection(s)
I20260925 04:39:06.231392 3539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
I20260925 04:39:06.240965 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 3539
I20260925 04:39:06.241120 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-0/wal/instance
I20260925 04:39:06.241428 392 external_mini_cluster.cc:1428] Setting key 04e08c9195584703806de627b907c4ff
Sep 25 04:39:06 dist-test-slave-dc95 krb5kdc[3432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311146, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:39:06.248008 3516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:06.233635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49639 (local address 127.0.98.62:39081)
0925 04:39:06.233820 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:06.233825 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:06.234855 (+ 1030us) server_negotiation.cc:407] Connection header received
0925 04:39:06.236295 (+ 1440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:06.236298 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:06.236354 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:06.236441 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:06.237972 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:06.238523 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:06.239901 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:06.240170 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:06.240267 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:06.243560 (+ 3293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:06.243562 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:06.243565 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:06.243591 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:06.245523 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:06.246273 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:06.246278 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:06.246279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:06.246385 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:06.246739 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:06.246742 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:06.246744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:06.246978 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:06.247122 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:06.247658 (+ 536us) server_negotiation.cc:299] Negotiation successful
0925 04:39:06.247828 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 04:39:06.248832 3670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:06.234045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39081 (local address 127.0.98.1:49639)
0925 04:39:06.234664 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:06.234714 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:06.236035 (+ 1321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:06.236834 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:06.236842 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:06.237264 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:06.237783 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:06.237800 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:06.238692 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:06.238695 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:06.239749 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:06.239761 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:06.240358 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:06.240362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:06.240532 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:06.241365 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:06.241389 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:06.243333 (+ 1944us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:39:06.245749 (+ 2416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:06.245758 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:06.245781 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:06.246150 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:06.246479 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:06.246481 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:06.246482 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:06.246649 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:06.247101 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:39:06.247107 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:39:06.247384 (+ 277us) client_negotiation.cc:769] Sending connection context
0925 04:39:06.247554 (+ 170us) client_negotiation.cc:240] Negotiation successful
0925 04:39:06.247902 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":128,"threads_started":1}
I20260925 04:39:06.250212 3668 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39081
I20260925 04:39:06.250466 3668 heartbeater.cc:464] Registering TS with master...
I20260925 04:39:06.251032 3668 heartbeater.cc:511] Master 127.0.98.62:39081 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 04:39:06.252802 3477 ts_manager.cc:194] Registered new tserver with Master: 9a9ce405fe5040a0bef7608fe3f23379 (127.0.98.1:34865)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260925 04:39:06.254194 3477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49639
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 04:39:06.271415 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39081
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:39:06.387539 3673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:39:06.387848 3673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:39:06.387893 3673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:39:06.392040 3673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:39:06.392124 3673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:39:06.392210 3673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 04:39:06.397787 3673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:39:06.398888 3673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:39:06.399928 3673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:39:06.410115 3679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:06.410115 3678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:06.410281 3673 server_base.cc:1034] running on GCE node
W20260925 04:39:06.410203 3681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:06.410816 3673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:39:06.411495 3673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:39:06.412683 3673 hybrid_clock.cc:648] HybridClock initialized: now 1790311146412661 us; error 39 us; skew 500 ppm
Sep 25 04:39:06 dist-test-slave-dc95 krb5kdc[3432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311146, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:39:06.415490 3673 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 04:39:06.416430 3673 webserver.cc:460] Webserver started at http://127.0.98.2:34373/ using document root <none> and password file <none>
I20260925 04:39:06.416973 3673 fs_manager.cc:359] Metadata directory not provided
I20260925 04:39:06.417042 3673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:39:06.417296 3673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:39:06.419409 3673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-1/data/instance:
uuid: "24d1680d3ff6401288180d8aad3546af"
format_stamp: "Formatted at 2026-09-25 04:39:06 on dist-test-slave-dc95"
server_key: "b1ef4d97e0ab13f46cb3a662cc188dc5"
server_key_iv: "73b4301426297c20ff20c77a1bfd2e12"
server_key_version: "encryptionkey@0"
I20260925 04:39:06.420008 3673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-1/wal/instance:
uuid: "24d1680d3ff6401288180d8aad3546af"
format_stamp: "Formatted at 2026-09-25 04:39:06 on dist-test-slave-dc95"
server_key: "b1ef4d97e0ab13f46cb3a662cc188dc5"
server_key_iv: "73b4301426297c20ff20c77a1bfd2e12"
server_key_version: "encryptionkey@0"
I20260925 04:39:06.424027 3673 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 04:39:06.426775 3688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:06.428297 3673 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:39:06.428423 3673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-1/wal
uuid: "24d1680d3ff6401288180d8aad3546af"
format_stamp: "Formatted at 2026-09-25 04:39:06 on dist-test-slave-dc95"
server_key: "b1ef4d97e0ab13f46cb3a662cc188dc5"
server_key_iv: "73b4301426297c20ff20c77a1bfd2e12"
server_key_version: "encryptionkey@0"
I20260925 04:39:06.428537 3673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:39:06.441666 3673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:39:06.445186 3673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:39:06.445448 3673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:39:06.446102 3673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:39:06.447029 3673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:39:06.447088 3673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:06.447131 3673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:39:06.447151 3673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:06.456128 3673 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42297
I20260925 04:39:06.456214 3801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42297 every 8 connection(s)
I20260925 04:39:06.457173 3673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
I20260925 04:39:06.458047 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 3673
I20260925 04:39:06.458163 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-1/wal/instance
I20260925 04:39:06.458411 392 external_mini_cluster.cc:1428] Setting key 9bc567bdca8139de46998c48e632a7ef
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Sep 25 04:39:06 dist-test-slave-dc95 krb5kdc[3432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311146, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:39:06.474422 3516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:06.459827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47551 (local address 127.0.98.62:39081)
0925 04:39:06.459942 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:06.459946 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:06.462031 (+ 2085us) server_negotiation.cc:407] Connection header received
0925 04:39:06.463464 (+ 1433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:06.463468 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:06.463546 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:06.463689 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:06.464936 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:06.465759 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:06.466979 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:06.467333 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:06.467450 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:06.470924 (+ 3474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:06.470926 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:06.470928 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:06.470954 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:06.472360 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:06.472884 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:06.472888 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:06.472889 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:06.472966 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:06.473288 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:06.473291 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:06.473293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:06.473511 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:06.473599 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:06.474125 (+ 526us) server_negotiation.cc:299] Negotiation successful
0925 04:39:06.474257 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 04:39:06.475317 3804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:06.459522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39081 (local address 127.0.98.2:47551)
0925 04:39:06.461835 (+ 2313us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:06.461890 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:06.463203 (+ 1313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:06.463888 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:06.463896 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:06.464271 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:06.464772 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:06.464787 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:06.465911 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:06.465914 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:06.466846 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:06.466858 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:06.467489 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:06.467492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:06.467684 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:06.468348 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:06.468365 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:06.470723 (+ 2358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:39:06.472485 (+ 1762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:06.472492 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:06.472511 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:06.472786 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:06.473049 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:06.473052 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:06.473054 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:06.473201 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:06.473617 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:39:06.473624 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:39:06.473886 (+ 262us) client_negotiation.cc:769] Sending connection context
0925 04:39:06.474062 (+ 176us) client_negotiation.cc:240] Negotiation successful
0925 04:39:06.474419 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2025,"thread_start_us":134,"threads_started":1}
I20260925 04:39:06.476670 3802 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39081
I20260925 04:39:06.476902 3802 heartbeater.cc:464] Registering TS with master...
I20260925 04:39:06.477401 3802 heartbeater.cc:511] Master 127.0.98.62:39081 requested a full tablet report, sending...
I20260925 04:39:06.478498 3477 ts_manager.cc:194] Registered new tserver with Master: 24d1680d3ff6401288180d8aad3546af (127.0.98.2:42297)
I20260925 04:39:06.479177 3477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47551
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 04:39:06.488317 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39081
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:39:06.599561 3807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:39:06.599895 3807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:39:06.599941 3807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:39:06.604068 3807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:39:06.604143 3807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:39:06.604206 3807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 04:39:06.609937 3807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:39:06.611030 3807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:39:06.612159 3807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:39:06.622670 3813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:06.622687 3807 server_base.cc:1034] running on GCE node
W20260925 04:39:06.622709 3812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:06.622704 3815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:06.623435 3807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:39:06.624281 3807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:39:06.625458 3807 hybrid_clock.cc:648] HybridClock initialized: now 1790311146625438 us; error 34 us; skew 500 ppm
Sep 25 04:39:06 dist-test-slave-dc95 krb5kdc[3432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311146, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:39:06.628458 3807 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 04:39:06.629453 3807 webserver.cc:460] Webserver started at http://127.0.98.3:41127/ using document root <none> and password file <none>
I20260925 04:39:06.630021 3807 fs_manager.cc:359] Metadata directory not provided
I20260925 04:39:06.630084 3807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:39:06.630287 3807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:39:06.632555 3807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-2/data/instance:
uuid: "45844b822b9c4648870640974db531ce"
format_stamp: "Formatted at 2026-09-25 04:39:06 on dist-test-slave-dc95"
server_key: "a6f1b67f9da80119311175482605e413"
server_key_iv: "8723faf7dd4d9590e057da697e001a98"
server_key_version: "encryptionkey@0"
I20260925 04:39:06.633143 3807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-2/wal/instance:
uuid: "45844b822b9c4648870640974db531ce"
format_stamp: "Formatted at 2026-09-25 04:39:06 on dist-test-slave-dc95"
server_key: "a6f1b67f9da80119311175482605e413"
server_key_iv: "8723faf7dd4d9590e057da697e001a98"
server_key_version: "encryptionkey@0"
I20260925 04:39:06.637084 3807 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:39:06.639950 3822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:06.641501 3807 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 04:39:06.641640 3807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-2/wal
uuid: "45844b822b9c4648870640974db531ce"
format_stamp: "Formatted at 2026-09-25 04:39:06 on dist-test-slave-dc95"
server_key: "a6f1b67f9da80119311175482605e413"
server_key_iv: "8723faf7dd4d9590e057da697e001a98"
server_key_version: "encryptionkey@0"
I20260925 04:39:06.641788 3807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:39:06.678831 3807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:39:06.682435 3807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:39:06.682626 3807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:39:06.683245 3807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:39:06.684226 3807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:39:06.684278 3807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:06.684332 3807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:39:06.684356 3807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:06.693040 3807 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35923
I20260925 04:39:06.693069 3935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35923 every 8 connection(s)
I20260925 04:39:06.694003 3807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
I20260925 04:39:06.695245 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 3807
I20260925 04:39:06.695356 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311123478858-392-0/minicluster-data/ts-2/wal/instance
I20260925 04:39:06.695755 392 external_mini_cluster.cc:1428] Setting key 8cdb9c55b7822b331b3b5f620c2fce39
Sep 25 04:39:06 dist-test-slave-dc95 krb5kdc[3432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311146, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:39:06.708244 3516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:06.696064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49299 (local address 127.0.98.62:39081)
0925 04:39:06.696226 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:06.696231 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:06.697341 (+ 1110us) server_negotiation.cc:407] Connection header received
0925 04:39:06.698761 (+ 1420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:06.698764 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:06.698819 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:06.698890 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:06.700108 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:06.700646 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:06.701805 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:06.702026 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:06.702114 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:06.704774 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:06.704777 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:06.704780 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:06.704808 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:06.706241 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:06.706771 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:06.706773 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:06.706774 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:06.706846 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:06.707128 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:06.707130 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:06.707132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:06.707307 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:06.707376 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:06.707988 (+ 612us) server_negotiation.cc:299] Negotiation successful
0925 04:39:06.708113 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 04:39:06.709365 3938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:06.696582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39081 (local address 127.0.98.3:49299)
0925 04:39:06.697203 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:06.697249 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:06.698548 (+ 1299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:06.699070 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:06.699084 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:06.699447 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:06.699961 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:06.699974 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:06.700771 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:06.700773 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:06.701677 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:06.701687 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:06.702177 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:06.702181 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:06.702351 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:06.702937 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:06.702953 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:06.704557 (+ 1604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:39:06.706364 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:06.706370 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:06.706386 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:06.706684 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:06.706914 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:06.706916 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:06.706917 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:06.707058 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:06.707383 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:39:06.707389 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:39:06.707721 (+ 332us) client_negotiation.cc:769] Sending connection context
0925 04:39:06.707906 (+ 185us) client_negotiation.cc:240] Negotiation successful
0925 04:39:06.708318 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":137,"threads_started":1}
I20260925 04:39:06.710469 3936 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39081
I20260925 04:39:06.710713 3936 heartbeater.cc:464] Registering TS with master...
I20260925 04:39:06.711246 3936 heartbeater.cc:511] Master 127.0.98.62:39081 requested a full tablet report, sending...
I20260925 04:39:06.712304 3477 ts_manager.cc:194] Registered new tserver with Master: 45844b822b9c4648870640974db531ce (127.0.98.3:35923)
I20260925 04:39:06.712925 3477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49299
I20260925 04:39:06.721904 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 04:39:06 dist-test-slave-dc95 krb5kdc[3432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311146, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 25 04:39:06 dist-test-slave-dc95 krb5kdc[3432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311146, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:39:06.747805 3516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:06.739074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59116 (local address 127.0.98.62:39081)
0925 04:39:06.739239 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:06.739243 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:06.739340 (+ 97us) server_negotiation.cc:407] Connection header received
0925 04:39:06.739478 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:06.739481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:06.739527 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:06.739597 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:06.740379 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:06.740916 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:06.741746 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:06.741959 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:06.742045 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:06.744613 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:06.744616 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:06.744618 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:06.744647 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:06.746107 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:06.746550 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:06.746553 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:06.746555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:06.746636 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:06.746893 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:06.746896 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:06.746898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:06.747140 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:06.747262 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:06.747493 (+ 231us) server_negotiation.cc:299] Negotiation successful
0925 04:39:06.747640 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260925 04:39:06.748193 3477 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:59116
I20260925 04:39:06.748965 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 3539
I20260925 04:39:06.753942 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 3673
I20260925 04:39:06.758837 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 3807
I20260925 04:39:06.763316 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 3445
2026-09-25T04:39:06Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2142 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:39:06 dist-test-slave-dc95 krb5kdc[3948](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:39:06 dist-test-slave-dc95 krb5kdc[3948](info): set up 2 sockets
Sep 25 04:39:06 dist-test-slave-dc95 krb5kdc[3948](info): commencing operation
krb5kdc: starting...
W20260925 04:39:07.731842 392 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/krb5kdc/test-user.keytab.
Sep 25 04:39:07 dist-test-slave-dc95 krb5kdc[3948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:39:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:39:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 04:39:07.859707 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43067
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43067
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:39:07.974282 3961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:39:07.974606 3961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:39:07.974665 3961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:39:07.978952 3961 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:39:07.979041 3961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:39:07.979068 3961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:39:07.979091 3961 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:39:07.979110 3961 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:39:07.984740 3961 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43067
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:39:07.985958 3961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:39:07.987056 3961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:39:07.995958 3966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:07.995975 3967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:07.996016 3961 server_base.cc:1034] running on GCE node
W20260925 04:39:07.995975 3969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:07.996757 3961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:39:07.997954 3961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:39:07.999157 3961 hybrid_clock.cc:648] HybridClock initialized: now 1790311147999138 us; error 35 us; skew 500 ppm
Sep 25 04:39:07 dist-test-slave-dc95 krb5kdc[3948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311147, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:39:08.002317 3961 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 04:39:08.003494 3961 webserver.cc:460] Webserver started at http://127.0.98.62:33789/ using document root <none> and password file <none>
I20260925 04:39:08.004153 3961 fs_manager.cc:359] Metadata directory not provided
I20260925 04:39:08.004231 3961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:39:08.004441 3961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:39:08.006978 3961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/minicluster-data/master-0/data/instance:
uuid: "ac4c6b39d31640898f15e5d8e89f3658"
format_stamp: "Formatted at 2026-09-25 04:39:08 on dist-test-slave-dc95"
server_key: "87b83bee20484e95f1f96e1dea26b4a1"
server_key_iv: "5c61bfe6be7b9faebc3263062afe9014"
server_key_version: "encryptionkey@0"
I20260925 04:39:08.007700 3961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/minicluster-data/master-0/wal/instance:
uuid: "ac4c6b39d31640898f15e5d8e89f3658"
format_stamp: "Formatted at 2026-09-25 04:39:08 on dist-test-slave-dc95"
server_key: "87b83bee20484e95f1f96e1dea26b4a1"
server_key_iv: "5c61bfe6be7b9faebc3263062afe9014"
server_key_version: "encryptionkey@0"
I20260925 04:39:08.012543 3961 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:39:08.015509 3976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:08.017083 3961 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:39:08.017233 3961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/minicluster-data/master-0/wal
uuid: "ac4c6b39d31640898f15e5d8e89f3658"
format_stamp: "Formatted at 2026-09-25 04:39:08 on dist-test-slave-dc95"
server_key: "87b83bee20484e95f1f96e1dea26b4a1"
server_key_iv: "5c61bfe6be7b9faebc3263062afe9014"
server_key_version: "encryptionkey@0"
I20260925 04:39:08.017367 3961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:39:08.029433 3961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:39:08.033094 3961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:39:08.033355 3961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:39:08.040877 3961 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43067
I20260925 04:39:08.040896 4028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43067 every 8 connection(s)
I20260925 04:39:08.041931 3961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
I20260925 04:39:08.045095 4029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:39:08.046708 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 3961
I20260925 04:39:08.046838 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/minicluster-data/master-0/wal/instance
I20260925 04:39:08.047087 392 external_mini_cluster.cc:1428] Setting key ad9211c40a6264bfdbd34437c00c9e8b
I20260925 04:39:08.052783 4029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac4c6b39d31640898f15e5d8e89f3658: Bootstrap starting.
I20260925 04:39:08.055681 4029 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac4c6b39d31640898f15e5d8e89f3658: Neither blocks nor log segments found. Creating new log.
I20260925 04:39:08.056533 4029 log.cc:824] T 00000000000000000000000000000000 P ac4c6b39d31640898f15e5d8e89f3658: Log is configured to *not* fsync() on all Append() calls
Sep 25 04:39:08 dist-test-slave-dc95 krb5kdc[3948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:39:08.058912 4029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac4c6b39d31640898f15e5d8e89f3658: No bootstrap required, opened a new log
I20260925 04:39:08.063704 4032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:08.048662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54636 (local address 127.0.98.62:43067)
0925 04:39:08.049168 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:08.049190 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:08.049236 (+ 46us) server_negotiation.cc:407] Connection header received
0925 04:39:08.050307 (+ 1071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:08.050337 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:08.050859 (+ 522us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:08.051325 (+ 466us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:08.052166 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:08.053601 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:08.054583 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:08.055263 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:08.055426 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:08.058236 (+ 2810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:08.058242 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:08.058260 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:08.058293 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:08.060270 (+ 1977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:08.060756 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:08.060762 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:08.060769 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:08.060909 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:08.061158 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:08.061162 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:08.061164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:08.061767 (+ 603us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:08.061964 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:08.062248 (+ 284us) server_negotiation.cc:299] Negotiation successful
0925 04:39:08.062584 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":120,"threads_started":1}
I20260925 04:39:08.063694 4029 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac4c6b39d31640898f15e5d8e89f3658 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac4c6b39d31640898f15e5d8e89f3658" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43067 } }
I20260925 04:39:08.064036 4029 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac4c6b39d31640898f15e5d8e89f3658 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:39:08.064101 4029 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac4c6b39d31640898f15e5d8e89f3658 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac4c6b39d31640898f15e5d8e89f3658, State: Initialized, Role: FOLLOWER
I20260925 04:39:08.064576 4029 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac4c6b39d31640898f15e5d8e89f3658 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac4c6b39d31640898f15e5d8e89f3658" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43067 } }
I20260925 04:39:08.064707 4029 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac4c6b39d31640898f15e5d8e89f3658 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:39:08.064769 4029 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac4c6b39d31640898f15e5d8e89f3658 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:39:08.064867 4029 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac4c6b39d31640898f15e5d8e89f3658 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:39:08.066088 4029 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac4c6b39d31640898f15e5d8e89f3658 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac4c6b39d31640898f15e5d8e89f3658" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43067 } }
I20260925 04:39:08.066460 4029 leader_election.cc:302] T 00000000000000000000000000000000 P ac4c6b39d31640898f15e5d8e89f3658 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac4c6b39d31640898f15e5d8e89f3658; no voters:
I20260925 04:39:08.066792 4029 leader_election.cc:288] T 00000000000000000000000000000000 P ac4c6b39d31640898f15e5d8e89f3658 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:39:08.066918 4034 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac4c6b39d31640898f15e5d8e89f3658 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:39:08.067111 4034 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac4c6b39d31640898f15e5d8e89f3658 [term 1 LEADER]: Becoming Leader. State: Replica: ac4c6b39d31640898f15e5d8e89f3658, State: Running, Role: LEADER
I20260925 04:39:08.067462 4034 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac4c6b39d31640898f15e5d8e89f3658 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac4c6b39d31640898f15e5d8e89f3658" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43067 } }
I20260925 04:39:08.068077 4029 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac4c6b39d31640898f15e5d8e89f3658 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:39:08.069238 4035 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac4c6b39d31640898f15e5d8e89f3658 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac4c6b39d31640898f15e5d8e89f3658" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac4c6b39d31640898f15e5d8e89f3658" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43067 } } }
I20260925 04:39:08.069240 4036 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac4c6b39d31640898f15e5d8e89f3658 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac4c6b39d31640898f15e5d8e89f3658. Latest consensus state: current_term: 1 leader_uuid: "ac4c6b39d31640898f15e5d8e89f3658" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac4c6b39d31640898f15e5d8e89f3658" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43067 } } }
I20260925 04:39:08.069440 4036 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac4c6b39d31640898f15e5d8e89f3658 [sys.catalog]: This master's current role is: LEADER
I20260925 04:39:08.069445 4035 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac4c6b39d31640898f15e5d8e89f3658 [sys.catalog]: This master's current role is: LEADER
I20260925 04:39:08.069818 4043 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:39:08.073096 4043 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:39:08.079123 4043 catalog_manager.cc:1347] Generated new cluster ID: 0491d037279e4b399a5c00c77537ef92
I20260925 04:39:08.079201 4043 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:39:08.100868 4043 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:39:08.101874 4043 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:39:08.107271 4043 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac4c6b39d31640898f15e5d8e89f3658: Generated new TSK 0
I20260925 04:39:08.108099 4043 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 04:39:08.152122 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43067
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:39:08.269325 4055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:39:08.269611 4055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:39:08.269675 4055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:39:08.273967 4055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:39:08.274057 4055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:39:08.274143 4055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 04:39:08.279709 4055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:39:08.280869 4055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:39:08.281951 4055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:39:08.292464 4063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:08.292436 4061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:08.292460 4060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:08.292876 4055 server_base.cc:1034] running on GCE node
I20260925 04:39:08.293327 4055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:39:08.294044 4055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:39:08.295243 4055 hybrid_clock.cc:648] HybridClock initialized: now 1790311148295222 us; error 35 us; skew 500 ppm
Sep 25 04:39:08 dist-test-slave-dc95 krb5kdc[3948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:39:08.298344 4055 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 04:39:08.299381 4055 webserver.cc:460] Webserver started at http://127.0.98.1:40055/ using document root <none> and password file <none>
I20260925 04:39:08.300016 4055 fs_manager.cc:359] Metadata directory not provided
I20260925 04:39:08.300088 4055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:39:08.300303 4055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:39:08.302507 4055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/minicluster-data/ts-0/data/instance:
uuid: "32adf44c1ce74ea1a8d91d45290fe805"
format_stamp: "Formatted at 2026-09-25 04:39:08 on dist-test-slave-dc95"
server_key: "744d281259b6a1f738b5f6ba1ee2795d"
server_key_iv: "8158c5c70a388c879d7dbfb5d611b02f"
server_key_version: "encryptionkey@0"
I20260925 04:39:08.303097 4055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/minicluster-data/ts-0/wal/instance:
uuid: "32adf44c1ce74ea1a8d91d45290fe805"
format_stamp: "Formatted at 2026-09-25 04:39:08 on dist-test-slave-dc95"
server_key: "744d281259b6a1f738b5f6ba1ee2795d"
server_key_iv: "8158c5c70a388c879d7dbfb5d611b02f"
server_key_version: "encryptionkey@0"
I20260925 04:39:08.307049 4055 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260925 04:39:08.309830 4070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:08.311197 4055 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:39:08.311323 4055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/minicluster-data/ts-0/wal
uuid: "32adf44c1ce74ea1a8d91d45290fe805"
format_stamp: "Formatted at 2026-09-25 04:39:08 on dist-test-slave-dc95"
server_key: "744d281259b6a1f738b5f6ba1ee2795d"
server_key_iv: "8158c5c70a388c879d7dbfb5d611b02f"
server_key_version: "encryptionkey@0"
I20260925 04:39:08.311450 4055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:39:08.338959 4055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:39:08.342610 4055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:39:08.342836 4055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:39:08.343462 4055 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:39:08.344398 4055 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:39:08.344447 4055 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:08.344501 4055 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:39:08.344530 4055 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:08.353309 4055 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34011
I20260925 04:39:08.353338 4183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34011 every 8 connection(s)
I20260925 04:39:08.354239 4055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
I20260925 04:39:08.359701 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 4055
I20260925 04:39:08.359838 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/minicluster-data/ts-0/wal/instance
I20260925 04:39:08.360134 392 external_mini_cluster.cc:1428] Setting key 5e670238739c8bdd129fdc9034c85377
Sep 25 04:39:08 dist-test-slave-dc95 krb5kdc[3948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 04:39:08.372083 4032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:08.356182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53437 (local address 127.0.98.62:43067)
0925 04:39:08.356317 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:08.356320 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:08.357315 (+ 995us) server_negotiation.cc:407] Connection header received
0925 04:39:08.358940 (+ 1625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:08.358944 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:08.359009 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:08.359088 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:08.360863 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:08.361634 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:08.363786 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:08.364067 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:08.364186 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:08.367517 (+ 3331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:08.367523 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:08.367526 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:08.367562 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:08.369358 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:08.370130 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:08.370133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:08.370135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:08.370231 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:08.370738 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:08.370741 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:08.370743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:08.370990 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:08.371095 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:08.371740 (+ 645us) server_negotiation.cc:299] Negotiation successful
0925 04:39:08.371932 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 04:39:08.372922 4186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:08.356632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43067 (local address 127.0.98.1:53437)
0925 04:39:08.357143 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:08.357190 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:08.358662 (+ 1472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:08.359302 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:08.359310 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:08.359883 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:08.360620 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:08.360642 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:08.361814 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:08.361817 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:08.363348 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:08.363365 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:08.364414 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:08.364418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:08.364662 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:08.365402 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:08.365426 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:08.367259 (+ 1833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:39:08.369549 (+ 2290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:08.369557 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:08.369581 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:08.369985 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:08.370346 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:08.370350 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:08.370352 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:08.370597 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:08.371113 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:39:08.371121 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:39:08.371441 (+ 320us) client_negotiation.cc:769] Sending connection context
0925 04:39:08.371652 (+ 211us) client_negotiation.cc:240] Negotiation successful
0925 04:39:08.371990 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":145,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260925 04:39:08.374514 4184 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43067
I20260925 04:39:08.374809 4184 heartbeater.cc:464] Registering TS with master...
I20260925 04:39:08.375480 4184 heartbeater.cc:511] Master 127.0.98.62:43067 requested a full tablet report, sending...
I20260925 04:39:08.377377 3993 ts_manager.cc:194] Registered new tserver with Master: 32adf44c1ce74ea1a8d91d45290fe805 (127.0.98.1:34011)
I20260925 04:39:08.378742 3993 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53437
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 04:39:08.391724 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43067
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:39:08.508430 4189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:39:08.508714 4189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:39:08.508759 4189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:39:08.513126 4189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:39:08.513219 4189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:39:08.513283 4189 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 04:39:08.519011 4189 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4189
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:39:08.520164 4189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:39:08.521240 4189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:39:08.531545 4197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:08.531545 4194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:08.531569 4195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:08.532085 4189 server_base.cc:1034] running on GCE node
I20260925 04:39:08.532516 4189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:39:08.533193 4189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:39:08.534382 4189 hybrid_clock.cc:648] HybridClock initialized: now 1790311148534367 us; error 28 us; skew 500 ppm
Sep 25 04:39:08 dist-test-slave-dc95 krb5kdc[3948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:39:08.537261 4189 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 04:39:08.538203 4189 webserver.cc:460] Webserver started at http://127.0.98.2:43465/ using document root <none> and password file <none>
I20260925 04:39:08.538757 4189 fs_manager.cc:359] Metadata directory not provided
I20260925 04:39:08.538823 4189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:39:08.539038 4189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:39:08.541173 4189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/minicluster-data/ts-1/data/instance:
uuid: "a04a0f8f1d1a41e8a0a47908d42a4f6c"
format_stamp: "Formatted at 2026-09-25 04:39:08 on dist-test-slave-dc95"
server_key: "23d2e5de0d91001e7909d0aee2ef6085"
server_key_iv: "edc33eeb1b7bf0253e5e15aafd21de16"
server_key_version: "encryptionkey@0"
I20260925 04:39:08.541739 4189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a04a0f8f1d1a41e8a0a47908d42a4f6c"
format_stamp: "Formatted at 2026-09-25 04:39:08 on dist-test-slave-dc95"
server_key: "23d2e5de0d91001e7909d0aee2ef6085"
server_key_iv: "edc33eeb1b7bf0253e5e15aafd21de16"
server_key_version: "encryptionkey@0"
I20260925 04:39:08.545707 4189 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 04:39:08.548485 4204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:08.549820 4189 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:39:08.549984 4189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/minicluster-data/ts-1/wal
uuid: "a04a0f8f1d1a41e8a0a47908d42a4f6c"
format_stamp: "Formatted at 2026-09-25 04:39:08 on dist-test-slave-dc95"
server_key: "23d2e5de0d91001e7909d0aee2ef6085"
server_key_iv: "edc33eeb1b7bf0253e5e15aafd21de16"
server_key_version: "encryptionkey@0"
I20260925 04:39:08.550096 4189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:39:08.565667 4189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:39:08.569540 4189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:39:08.569744 4189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:39:08.570363 4189 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:39:08.571228 4189 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:39:08.571285 4189 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:08.571354 4189 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:39:08.571379 4189 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:08.580175 4189 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42429
I20260925 04:39:08.580240 4317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42429 every 8 connection(s)
I20260925 04:39:08.581123 4189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
I20260925 04:39:08.588753 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 4189
I20260925 04:39:08.588888 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/minicluster-data/ts-1/wal/instance
I20260925 04:39:08.589147 392 external_mini_cluster.cc:1428] Setting key 09f8cff427bb2a345323fa84c8c54aaf
Sep 25 04:39:08 dist-test-slave-dc95 krb5kdc[3948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:39:08.596015 4032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:08.583074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40473 (local address 127.0.98.62:43067)
0925 04:39:08.583209 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:08.583212 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:08.584184 (+ 972us) server_negotiation.cc:407] Connection header received
0925 04:39:08.585606 (+ 1422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:08.585609 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:08.585661 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:08.585740 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:08.587000 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:08.587529 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:08.588893 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:08.589194 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:08.589294 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:08.592265 (+ 2971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:08.592268 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:08.592270 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:08.592294 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:08.593626 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:08.594250 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:08.594254 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:08.594255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:08.594342 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:08.594710 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:08.594713 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:08.594714 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:08.594950 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:08.595030 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:08.595734 (+ 704us) server_negotiation.cc:299] Negotiation successful
0925 04:39:08.595846 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 04:39:08.597193 4320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:08.583493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43067 (local address 127.0.98.2:40473)
0925 04:39:08.584016 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:08.584063 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:08.585323 (+ 1260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:08.585902 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:08.585910 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:08.586286 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:08.586825 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:08.586841 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:08.587702 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:08.587705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:08.588755 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:08.588767 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:08.589371 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:08.589375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:08.589578 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:08.590181 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:08.590203 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:08.592082 (+ 1879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:39:08.593740 (+ 1658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:08.593746 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:08.593765 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:08.594111 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:08.594428 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:08.594431 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:08.594433 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:08.594608 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:08.595057 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:39:08.595064 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:39:08.595424 (+ 360us) client_negotiation.cc:769] Sending connection context
0925 04:39:08.595663 (+ 239us) client_negotiation.cc:240] Negotiation successful
0925 04:39:08.596056 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":127,"threads_started":1}
I20260925 04:39:08.598488 4318 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43067
I20260925 04:39:08.598748 4318 heartbeater.cc:464] Registering TS with master...
I20260925 04:39:08.599242 4318 heartbeater.cc:511] Master 127.0.98.62:43067 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260925 04:39:08.600330 3993 ts_manager.cc:194] Registered new tserver with Master: a04a0f8f1d1a41e8a0a47908d42a4f6c (127.0.98.2:42429)
I20260925 04:39:08.600962 3993 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40473
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 04:39:08.619730 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43067
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:39:08.732576 4323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:39:08.732878 4323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:39:08.732939 4323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:39:08.737298 4323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:39:08.737387 4323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:39:08.737474 4323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 04:39:08.743103 4323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:39:08.744354 4323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:39:08.745607 4323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:39:08.755890 4328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:08.756023 4323 server_base.cc:1034] running on GCE node
W20260925 04:39:08.755882 4329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:08.755970 4331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:08.756601 4323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:39:08.757310 4323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:39:08.758489 4323 hybrid_clock.cc:648] HybridClock initialized: now 1790311148758451 us; error 50 us; skew 500 ppm
Sep 25 04:39:08 dist-test-slave-dc95 krb5kdc[3948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:39:08.761431 4323 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 04:39:08.762343 4323 webserver.cc:460] Webserver started at http://127.0.98.3:43915/ using document root <none> and password file <none>
I20260925 04:39:08.762946 4323 fs_manager.cc:359] Metadata directory not provided
I20260925 04:39:08.763017 4323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:39:08.763226 4323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:39:08.765439 4323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/minicluster-data/ts-2/data/instance:
uuid: "c4ab9a7a9d3e4190986cf0bd1b4b38ed"
format_stamp: "Formatted at 2026-09-25 04:39:08 on dist-test-slave-dc95"
server_key: "bacbad34fbf0ed28f981037f6e787d8d"
server_key_iv: "60509654fb03f838f30907f0d1e8a1f6"
server_key_version: "encryptionkey@0"
I20260925 04:39:08.766083 4323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c4ab9a7a9d3e4190986cf0bd1b4b38ed"
format_stamp: "Formatted at 2026-09-25 04:39:08 on dist-test-slave-dc95"
server_key: "bacbad34fbf0ed28f981037f6e787d8d"
server_key_iv: "60509654fb03f838f30907f0d1e8a1f6"
server_key_version: "encryptionkey@0"
I20260925 04:39:08.770215 4323 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260925 04:39:08.772903 4338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:08.774290 4323 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 04:39:08.774438 4323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/minicluster-data/ts-2/wal
uuid: "c4ab9a7a9d3e4190986cf0bd1b4b38ed"
format_stamp: "Formatted at 2026-09-25 04:39:08 on dist-test-slave-dc95"
server_key: "bacbad34fbf0ed28f981037f6e787d8d"
server_key_iv: "60509654fb03f838f30907f0d1e8a1f6"
server_key_version: "encryptionkey@0"
I20260925 04:39:08.774552 4323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:39:08.792824 4323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:39:08.796382 4323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:39:08.796586 4323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:39:08.797192 4323 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:39:08.798043 4323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:39:08.798091 4323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:08.798146 4323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:39:08.798169 4323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:08.806941 4323 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43505
I20260925 04:39:08.807013 4451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43505 every 8 connection(s)
I20260925 04:39:08.807979 4323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
I20260925 04:39:08.816624 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 4323
I20260925 04:39:08.816748 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311123478858-392-0/minicluster-data/ts-2/wal/instance
I20260925 04:39:08.817020 392 external_mini_cluster.cc:1428] Setting key 90e1871ed1dac702d3ab2955445257a7
Sep 25 04:39:08 dist-test-slave-dc95 krb5kdc[3948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:39:08.823405 4032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:08.809992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34693 (local address 127.0.98.62:43067)
0925 04:39:08.810137 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:08.810141 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:08.811159 (+ 1018us) server_negotiation.cc:407] Connection header received
0925 04:39:08.812686 (+ 1527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:08.812689 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:08.812752 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:08.812828 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:08.814172 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:08.814705 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:08.816067 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:08.816286 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:08.816391 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:08.819413 (+ 3022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:08.819417 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:08.819420 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:08.819449 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:08.821066 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:08.821673 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:08.821676 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:08.821678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:08.821770 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:08.822154 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:08.822158 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:08.822160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:08.822371 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:08.822462 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:08.823094 (+ 632us) server_negotiation.cc:299] Negotiation successful
0925 04:39:08.823232 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 04:39:08.824290 4454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:08.810432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43067 (local address 127.0.98.3:34693)
0925 04:39:08.810969 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:08.811020 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:08.812419 (+ 1399us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:08.813047 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:08.813055 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:08.813462 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:08.813996 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:08.814012 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:08.814841 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:08.814844 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:08.815926 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:08.815938 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:08.816481 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:08.816485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:08.816667 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:08.817341 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:08.817364 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:08.819171 (+ 1807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:39:08.821213 (+ 2042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:08.821220 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:08.821237 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:08.821548 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:08.821859 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:08.821862 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:08.821863 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:08.822042 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:08.822478 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:39:08.822487 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:39:08.822795 (+ 308us) client_negotiation.cc:769] Sending connection context
0925 04:39:08.822976 (+ 181us) client_negotiation.cc:240] Negotiation successful
0925 04:39:08.823356 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":132,"threads_started":1}
I20260925 04:39:08.825525 4452 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43067
I20260925 04:39:08.825783 4452 heartbeater.cc:464] Registering TS with master...
I20260925 04:39:08.826308 4452 heartbeater.cc:511] Master 127.0.98.62:43067 requested a full tablet report, sending...
I20260925 04:39:08.827445 3993 ts_manager.cc:194] Registered new tserver with Master: c4ab9a7a9d3e4190986cf0bd1b4b38ed (127.0.98.3:43505)
I20260925 04:39:08.828054 3993 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34693
I20260925 04:39:08.831497 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 04:39:08 dist-test-slave-dc95 krb5kdc[3948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 04:39:08 dist-test-slave-dc95 krb5kdc[3948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 04:39:08.857816 4459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:08.847634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56802 (local address 127.0.98.1:34011)
0925 04:39:08.847914 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:08.847930 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:08.847956 (+ 26us) server_negotiation.cc:407] Connection header received
0925 04:39:08.848097 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:08.848104 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:08.848304 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:08.848455 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:08.849148 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:08.849935 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:08.850813 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:08.851199 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:08.851301 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:08.853926 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:08.853943 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:08.853960 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:08.853992 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:08.855826 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:08.856310 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:08.856316 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:08.856317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:08.856402 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:08.856767 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:08.856771 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:08.856773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:08.857072 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:08.857283 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:08.857519 (+ 236us) server_negotiation.cc:299] Negotiation successful
0925 04:39:08.857660 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":81,"threads_started":1}
Sep 25 04:39:08 dist-test-slave-dc95 krb5kdc[3948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:39:08.868881 4032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:08.860821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55548 (local address 127.0.98.62:43067)
0925 04:39:08.860958 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:08.860962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:08.860974 (+ 12us) server_negotiation.cc:407] Connection header received
0925 04:39:08.861050 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:08.861052 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:08.861093 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:08.861147 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:08.861718 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:08.862245 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:08.863047 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:08.863258 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:08.863348 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:08.865739 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:08.865742 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:08.865744 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:08.865770 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:08.867103 (+ 1333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:08.867499 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:08.867501 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:08.867503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:08.867575 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:08.867917 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:08.867921 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:08.867967 (+ 46us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:08.868216 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:08.868328 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:08.868595 (+ 267us) server_negotiation.cc:299] Negotiation successful
0925 04:39:08.868718 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260925 04:39:08.869313 3993 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55548
I20260925 04:39:08.870015 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 4055
I20260925 04:39:08.874996 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 4189
I20260925 04:39:08.879678 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 4323
I20260925 04:39:08.884192 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 3961
2026-09-25T04:39:08Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2120 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:39:08 dist-test-slave-dc95 krb5kdc[4464](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:39:08 dist-test-slave-dc95 krb5kdc[4464](info): set up 2 sockets
Sep 25 04:39:08 dist-test-slave-dc95 krb5kdc[4464](info): commencing operation
krb5kdc: starting...
W20260925 04:39:09.847599 392 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/krb5kdc/test-user.keytab.
Sep 25 04:39:09 dist-test-slave-dc95 krb5kdc[4464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311149, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:39:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:39:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 04:39:09.984654 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37853
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37853
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:39:10.099400 4477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:39:10.099773 4477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:39:10.099843 4477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:39:10.104123 4477 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:39:10.104206 4477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:39:10.104234 4477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:39:10.104255 4477 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:39:10.104275 4477 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:39:10.110023 4477 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:39:10.111179 4477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:39:10.112283 4477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:39:10.120645 4482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:10.120690 4483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:10.120648 4485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:10.120893 4477 server_base.cc:1034] running on GCE node
I20260925 04:39:10.121603 4477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:39:10.122833 4477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:39:10.124002 4477 hybrid_clock.cc:648] HybridClock initialized: now 1790311150123991 us; error 37 us; skew 500 ppm
Sep 25 04:39:10 dist-test-slave-dc95 krb5kdc[4464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311150, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:39:10.127060 4477 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 04:39:10.127986 4477 fs_manager.cc:359] Metadata directory not provided
I20260925 04:39:10.128077 4477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:39:10.128298 4477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:39:10.130515 4477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/minicluster-data/master-0/data/instance:
uuid: "06cd63cb831f47ceab62dc54f45ef0c7"
format_stamp: "Formatted at 2026-09-25 04:39:10 on dist-test-slave-dc95"
server_key: "531e90c176f4ddd018c8659b08fd5026"
server_key_iv: "1a8c0719bfc20ecf25c766d42c81ba85"
server_key_version: "encryptionkey@0"
I20260925 04:39:10.131155 4477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/minicluster-data/master-0/wal/instance:
uuid: "06cd63cb831f47ceab62dc54f45ef0c7"
format_stamp: "Formatted at 2026-09-25 04:39:10 on dist-test-slave-dc95"
server_key: "531e90c176f4ddd018c8659b08fd5026"
server_key_iv: "1a8c0719bfc20ecf25c766d42c81ba85"
server_key_version: "encryptionkey@0"
I20260925 04:39:10.135181 4477 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260925 04:39:10.138121 4491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:10.139555 4477 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:39:10.139745 4477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/minicluster-data/master-0/wal
uuid: "06cd63cb831f47ceab62dc54f45ef0c7"
format_stamp: "Formatted at 2026-09-25 04:39:10 on dist-test-slave-dc95"
server_key: "531e90c176f4ddd018c8659b08fd5026"
server_key_iv: "1a8c0719bfc20ecf25c766d42c81ba85"
server_key_version: "encryptionkey@0"
I20260925 04:39:10.139864 4477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:39:10.153134 4477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:39:10.156661 4477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:39:10.156867 4477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:39:10.164254 4477 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37853
I20260925 04:39:10.164290 4543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37853 every 8 connection(s)
I20260925 04:39:10.164954 4477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
I20260925 04:39:10.168120 4544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:39:10.171558 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 4477
I20260925 04:39:10.171818 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/minicluster-data/master-0/wal/instance
I20260925 04:39:10.172200 392 external_mini_cluster.cc:1428] Setting key 7934baeb5cdef7fa32e24fb122d77a0c
I20260925 04:39:10.175071 4544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06cd63cb831f47ceab62dc54f45ef0c7: Bootstrap starting.
I20260925 04:39:10.177734 4544 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06cd63cb831f47ceab62dc54f45ef0c7: Neither blocks nor log segments found. Creating new log.
I20260925 04:39:10.178516 4544 log.cc:824] T 00000000000000000000000000000000 P 06cd63cb831f47ceab62dc54f45ef0c7: Log is configured to *not* fsync() on all Append() calls
I20260925 04:39:10.180770 4544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06cd63cb831f47ceab62dc54f45ef0c7: No bootstrap required, opened a new log
Sep 25 04:39:10 dist-test-slave-dc95 krb5kdc[4464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311149, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:39:10.184526 4544 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06cd63cb831f47ceab62dc54f45ef0c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06cd63cb831f47ceab62dc54f45ef0c7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37853 } }
I20260925 04:39:10.184813 4544 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06cd63cb831f47ceab62dc54f45ef0c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:39:10.184847 4544 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06cd63cb831f47ceab62dc54f45ef0c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06cd63cb831f47ceab62dc54f45ef0c7, State: Initialized, Role: FOLLOWER
I20260925 04:39:10.185395 4544 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06cd63cb831f47ceab62dc54f45ef0c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06cd63cb831f47ceab62dc54f45ef0c7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37853 } }
I20260925 04:39:10.185534 4544 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06cd63cb831f47ceab62dc54f45ef0c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:39:10.185595 4544 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06cd63cb831f47ceab62dc54f45ef0c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:39:10.185695 4544 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06cd63cb831f47ceab62dc54f45ef0c7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:39:10.186657 4544 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06cd63cb831f47ceab62dc54f45ef0c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06cd63cb831f47ceab62dc54f45ef0c7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37853 } }
I20260925 04:39:10.187050 4544 leader_election.cc:302] T 00000000000000000000000000000000 P 06cd63cb831f47ceab62dc54f45ef0c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06cd63cb831f47ceab62dc54f45ef0c7; no voters:
I20260925 04:39:10.187417 4544 leader_election.cc:288] T 00000000000000000000000000000000 P 06cd63cb831f47ceab62dc54f45ef0c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:39:10.187532 4549 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06cd63cb831f47ceab62dc54f45ef0c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:39:10.187770 4549 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06cd63cb831f47ceab62dc54f45ef0c7 [term 1 LEADER]: Becoming Leader. State: Replica: 06cd63cb831f47ceab62dc54f45ef0c7, State: Running, Role: LEADER
I20260925 04:39:10.188174 4549 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06cd63cb831f47ceab62dc54f45ef0c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06cd63cb831f47ceab62dc54f45ef0c7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37853 } }
I20260925 04:39:10.188242 4547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:10.173742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41026 (local address 127.0.98.62:37853)
0925 04:39:10.174172 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:10.174262 (+ 90us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:10.174303 (+ 41us) server_negotiation.cc:407] Connection header received
0925 04:39:10.174969 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:10.174987 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:10.175460 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:10.175855 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:10.176731 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:10.177884 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:10.178827 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:10.179346 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:10.179502 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:10.182257 (+ 2755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:10.182263 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:10.182278 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:10.182313 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:10.184863 (+ 2550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:10.185367 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:10.185373 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:10.185378 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:10.185533 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:10.185842 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:10.185846 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:10.185847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:10.186302 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:10.186506 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:10.186812 (+ 306us) server_negotiation.cc:299] Negotiation successful
0925 04:39:10.187125 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":117,"threads_started":1}
I20260925 04:39:10.188719 4544 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06cd63cb831f47ceab62dc54f45ef0c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:39:10.191191 4551 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06cd63cb831f47ceab62dc54f45ef0c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06cd63cb831f47ceab62dc54f45ef0c7. Latest consensus state: current_term: 1 leader_uuid: "06cd63cb831f47ceab62dc54f45ef0c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06cd63cb831f47ceab62dc54f45ef0c7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37853 } } }
I20260925 04:39:10.191354 4551 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06cd63cb831f47ceab62dc54f45ef0c7 [sys.catalog]: This master's current role is: LEADER
I20260925 04:39:10.191502 4550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06cd63cb831f47ceab62dc54f45ef0c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06cd63cb831f47ceab62dc54f45ef0c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06cd63cb831f47ceab62dc54f45ef0c7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37853 } } }
I20260925 04:39:10.191639 4550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06cd63cb831f47ceab62dc54f45ef0c7 [sys.catalog]: This master's current role is: LEADER
W20260925 04:39:10.194115 4563 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 06cd63cb831f47ceab62dc54f45ef0c7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 04:39:10.194205 4563 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 04:39:10.194285 4565 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:39:10.195137 4565 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:39:10.201535 4565 catalog_manager.cc:1347] Generated new cluster ID: c41b85b432c744c88bff576d440dbbba
I20260925 04:39:10.201613 4565 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:39:10.212075 4565 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:39:10.213106 4565 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:39:10.220151 4565 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06cd63cb831f47ceab62dc54f45ef0c7: Generated new TSK 0
I20260925 04:39:10.220937 4565 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 04:39:10.254428 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37853
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:39:10.392005 4570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:39:10.392318 4570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:39:10.392365 4570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:39:10.396605 4570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:39:10.396683 4570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:39:10.396754 4570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 04:39:10.402350 4570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:39:10.403512 4570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:39:10.404644 4570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:39:10.414892 4575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:10.414947 4570 server_base.cc:1034] running on GCE node
W20260925 04:39:10.414901 4576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:10.414901 4578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:10.415517 4570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:39:10.416188 4570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:39:10.417351 4570 hybrid_clock.cc:648] HybridClock initialized: now 1790311150417337 us; error 30 us; skew 500 ppm
Sep 25 04:39:10 dist-test-slave-dc95 krb5kdc[4464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311150, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:39:10.420154 4570 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 04:39:10.420923 4570 fs_manager.cc:359] Metadata directory not provided
I20260925 04:39:10.420998 4570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:39:10.421197 4570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:39:10.423295 4570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/minicluster-data/ts-0/data/instance:
uuid: "681076b637134dc89507bd7b4da94664"
format_stamp: "Formatted at 2026-09-25 04:39:10 on dist-test-slave-dc95"
server_key: "e26851c54270d575fc27f18ddd997051"
server_key_iv: "d6c0ee7a23da2abfcc76e851b110c91a"
server_key_version: "encryptionkey@0"
I20260925 04:39:10.423887 4570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/minicluster-data/ts-0/wal/instance:
uuid: "681076b637134dc89507bd7b4da94664"
format_stamp: "Formatted at 2026-09-25 04:39:10 on dist-test-slave-dc95"
server_key: "e26851c54270d575fc27f18ddd997051"
server_key_iv: "d6c0ee7a23da2abfcc76e851b110c91a"
server_key_version: "encryptionkey@0"
I20260925 04:39:10.427673 4570 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 04:39:10.430346 4584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:10.431675 4570 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:39:10.431811 4570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/minicluster-data/ts-0/wal
uuid: "681076b637134dc89507bd7b4da94664"
format_stamp: "Formatted at 2026-09-25 04:39:10 on dist-test-slave-dc95"
server_key: "e26851c54270d575fc27f18ddd997051"
server_key_iv: "d6c0ee7a23da2abfcc76e851b110c91a"
server_key_version: "encryptionkey@0"
I20260925 04:39:10.431917 4570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:39:10.450958 4570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:39:10.454465 4570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:39:10.454674 4570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:39:10.455343 4570 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:39:10.456254 4570 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:39:10.456305 4570 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:10.456365 4570 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:39:10.456395 4570 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:10.465327 4570 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44215
I20260925 04:39:10.465346 4697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44215 every 8 connection(s)
I20260925 04:39:10.465971 4570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
I20260925 04:39:10.472091 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 4570
I20260925 04:39:10.472198 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/minicluster-data/ts-0/wal/instance
I20260925 04:39:10.472465 392 external_mini_cluster.cc:1428] Setting key c8427bef685aff5fd60ddba7f7b35a7b
Sep 25 04:39:10 dist-test-slave-dc95 krb5kdc[4464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311150, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:39:10.480674 4547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:10.467766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38831 (local address 127.0.98.62:37853)
0925 04:39:10.467926 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:10.467930 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:10.468815 (+ 885us) server_negotiation.cc:407] Connection header received
0925 04:39:10.470423 (+ 1608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:10.470427 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:10.470490 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:10.470561 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:10.471873 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:10.472473 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:10.473897 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:10.474124 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:10.474210 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:10.476696 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:10.476700 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:10.476704 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:10.476733 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:10.478347 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:10.478914 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:10.478918 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:10.478920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:10.479015 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:10.479435 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:10.479439 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:10.479441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:10.479721 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:10.479825 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:10.480354 (+ 529us) server_negotiation.cc:299] Negotiation successful
0925 04:39:10.480488 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 04:39:10.481467 4700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:10.468172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37853 (local address 127.0.98.1:38831)
0925 04:39:10.468640 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:10.468688 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:10.470159 (+ 1471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:10.470772 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:10.470779 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:10.471169 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:10.471724 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:10.471741 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:10.472608 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:10.472612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:10.473792 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:10.473803 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:10.474266 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:10.474270 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:10.474446 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:10.475020 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:10.475037 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:10.476508 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:39:10.478478 (+ 1970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:10.478484 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:10.478499 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:10.478797 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:10.479129 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:10.479132 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:10.479133 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:10.479331 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:10.479816 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:39:10.479821 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:39:10.480085 (+ 264us) client_negotiation.cc:769] Sending connection context
0925 04:39:10.480275 (+ 190us) client_negotiation.cc:240] Negotiation successful
0925 04:39:10.480552 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":110,"threads_started":1}
I20260925 04:39:10.482772 4698 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37853
I20260925 04:39:10.483033 4698 heartbeater.cc:464] Registering TS with master...
I20260925 04:39:10.483644 4698 heartbeater.cc:511] Master 127.0.98.62:37853 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260925 04:39:10.485381 4503 ts_manager.cc:194] Registered new tserver with Master: 681076b637134dc89507bd7b4da94664 (127.0.98.1:44215)
I20260925 04:39:10.486591 4503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38831
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 04:39:10.500552 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37853
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:39:10.613191 4703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:39:10.613499 4703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:39:10.613564 4703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:39:10.617808 4703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:39:10.617889 4703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:39:10.617983 4703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 04:39:10.623517 4703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:39:10.624696 4703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:39:10.625757 4703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:39:10.635783 4709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:10.635783 4708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:10.635797 4711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:10.636245 4703 server_base.cc:1034] running on GCE node
I20260925 04:39:10.636662 4703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:39:10.637385 4703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:39:10.638577 4703 hybrid_clock.cc:648] HybridClock initialized: now 1790311150638549 us; error 45 us; skew 500 ppm
Sep 25 04:39:10 dist-test-slave-dc95 krb5kdc[4464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311150, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:39:10.641573 4703 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 04:39:10.642426 4703 fs_manager.cc:359] Metadata directory not provided
I20260925 04:39:10.642500 4703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:39:10.642705 4703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:39:10.645022 4703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/minicluster-data/ts-1/data/instance:
uuid: "89eaea2b54794ba59550154e057c4627"
format_stamp: "Formatted at 2026-09-25 04:39:10 on dist-test-slave-dc95"
server_key: "28ff9505d623516c1e915384183cd473"
server_key_iv: "6e66276487c19daf8b4cfb5a70fd58cd"
server_key_version: "encryptionkey@0"
I20260925 04:39:10.645622 4703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/minicluster-data/ts-1/wal/instance:
uuid: "89eaea2b54794ba59550154e057c4627"
format_stamp: "Formatted at 2026-09-25 04:39:10 on dist-test-slave-dc95"
server_key: "28ff9505d623516c1e915384183cd473"
server_key_iv: "6e66276487c19daf8b4cfb5a70fd58cd"
server_key_version: "encryptionkey@0"
I20260925 04:39:10.649814 4703 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 04:39:10.652680 4717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:10.654064 4703 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:39:10.654227 4703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/minicluster-data/ts-1/wal
uuid: "89eaea2b54794ba59550154e057c4627"
format_stamp: "Formatted at 2026-09-25 04:39:10 on dist-test-slave-dc95"
server_key: "28ff9505d623516c1e915384183cd473"
server_key_iv: "6e66276487c19daf8b4cfb5a70fd58cd"
server_key_version: "encryptionkey@0"
I20260925 04:39:10.654361 4703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:39:10.703653 4703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:39:10.707283 4703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:39:10.707480 4703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:39:10.708138 4703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:39:10.708984 4703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:39:10.709033 4703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:10.709074 4703 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:39:10.709093 4703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:10.717792 4703 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43171
I20260925 04:39:10.717819 4830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43171 every 8 connection(s)
I20260925 04:39:10.718415 4703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
I20260925 04:39:10.728029 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 4703
I20260925 04:39:10.728130 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/minicluster-data/ts-1/wal/instance
I20260925 04:39:10.728406 392 external_mini_cluster.cc:1428] Setting key 02d5bf2ffc097b4634bb79ae3216fe59
Sep 25 04:39:10 dist-test-slave-dc95 krb5kdc[4464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311150, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:39:10.733297 4547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:10.720238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45475 (local address 127.0.98.62:37853)
0925 04:39:10.720375 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:10.720379 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:10.721382 (+ 1003us) server_negotiation.cc:407] Connection header received
0925 04:39:10.722902 (+ 1520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:10.722906 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:10.722969 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:10.723045 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:10.724439 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:10.724968 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:10.726147 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:10.726372 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:10.726463 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:10.729197 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:10.729202 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:10.729204 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:10.729233 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:10.730810 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:10.731393 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:10.731397 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:10.731399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:10.731499 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:10.731968 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:10.731971 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:10.731973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:10.732205 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:10.732287 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:10.733032 (+ 745us) server_negotiation.cc:299] Negotiation successful
0925 04:39:10.733164 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 04:39:10.734586 4833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:10.720744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37853 (local address 127.0.98.2:45475)
0925 04:39:10.721223 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:10.721268 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:10.722635 (+ 1367us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:10.723249 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:10.723257 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:10.723754 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:10.724282 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:10.724297 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:10.725094 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:10.725096 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:10.726011 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:10.726021 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:10.726524 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:10.726528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:10.726713 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:10.727321 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:10.727339 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:10.729016 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:39:10.730955 (+ 1939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:10.730961 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:10.730983 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:10.731285 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:10.731594 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:10.731598 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:10.731600 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:10.731858 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:10.732305 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:39:10.732312 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:39:10.732687 (+ 375us) client_negotiation.cc:769] Sending connection context
0925 04:39:10.732929 (+ 242us) client_negotiation.cc:240] Negotiation successful
0925 04:39:10.733311 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":107,"threads_started":1}
I20260925 04:39:10.736074 4831 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37853
I20260925 04:39:10.736374 4831 heartbeater.cc:464] Registering TS with master...
I20260925 04:39:10.736886 4831 heartbeater.cc:511] Master 127.0.98.62:37853 requested a full tablet report, sending...
I20260925 04:39:10.738023 4503 ts_manager.cc:194] Registered new tserver with Master: 89eaea2b54794ba59550154e057c4627 (127.0.98.2:43171)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260925 04:39:10.738621 4503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45475
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 04:39:10.759027 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37853
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:39:10.886435 4836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:39:10.886777 4836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:39:10.886849 4836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:39:10.891418 4836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:39:10.891502 4836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:39:10.891588 4836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 04:39:10.897333 4836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4836
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:39:10.898476 4836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:39:10.899586 4836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:39:10.909430 4844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:10.909426 4842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:10.909420 4841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:10.909914 4836 server_base.cc:1034] running on GCE node
I20260925 04:39:10.910383 4836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:39:10.911063 4836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:39:10.912216 4836 hybrid_clock.cc:648] HybridClock initialized: now 1790311150912190 us; error 41 us; skew 500 ppm
Sep 25 04:39:10 dist-test-slave-dc95 krb5kdc[4464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311150, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:39:10.915102 4836 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 04:39:10.915906 4836 fs_manager.cc:359] Metadata directory not provided
I20260925 04:39:10.915973 4836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:39:10.916146 4836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:39:10.918241 4836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/minicluster-data/ts-2/data/instance:
uuid: "4a8ce6d32ddc4a4aa99777adc2733cfa"
format_stamp: "Formatted at 2026-09-25 04:39:10 on dist-test-slave-dc95"
server_key: "d6f82e373a705ec5284f96d5472254f7"
server_key_iv: "7cbe5b68e2472a0b74c961ced485d4c3"
server_key_version: "encryptionkey@0"
I20260925 04:39:10.918779 4836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4a8ce6d32ddc4a4aa99777adc2733cfa"
format_stamp: "Formatted at 2026-09-25 04:39:10 on dist-test-slave-dc95"
server_key: "d6f82e373a705ec5284f96d5472254f7"
server_key_iv: "7cbe5b68e2472a0b74c961ced485d4c3"
server_key_version: "encryptionkey@0"
I20260925 04:39:10.922631 4836 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 04:39:10.925369 4850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:10.926867 4836 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 04:39:10.926993 4836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/minicluster-data/ts-2/wal
uuid: "4a8ce6d32ddc4a4aa99777adc2733cfa"
format_stamp: "Formatted at 2026-09-25 04:39:10 on dist-test-slave-dc95"
server_key: "d6f82e373a705ec5284f96d5472254f7"
server_key_iv: "7cbe5b68e2472a0b74c961ced485d4c3"
server_key_version: "encryptionkey@0"
I20260925 04:39:10.927102 4836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:39:10.949873 4836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:39:10.953392 4836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:39:10.953562 4836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:39:10.954154 4836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:39:10.955014 4836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:39:10.955070 4836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:10.955111 4836 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:39:10.955130 4836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:10.964690 4836 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38797
I20260925 04:39:10.964706 4963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38797 every 8 connection(s)
I20260925 04:39:10.965333 4836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
I20260925 04:39:10.966094 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 4836
I20260925 04:39:10.966231 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/minicluster-data/ts-2/wal/instance
I20260925 04:39:10.966547 392 external_mini_cluster.cc:1428] Setting key fcd2041d105a74ef0265bcff6d087edd
Sep 25 04:39:10 dist-test-slave-dc95 krb5kdc[4464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311150, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:39:10.980826 4547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:10.967544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33147 (local address 127.0.98.62:37853)
0925 04:39:10.967720 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:10.967724 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:10.968947 (+ 1223us) server_negotiation.cc:407] Connection header received
0925 04:39:10.970335 (+ 1388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:10.970338 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:10.970400 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:10.970472 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:10.971817 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:10.972365 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:10.973579 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:10.973811 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:10.973902 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:10.976677 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:10.976680 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:10.976683 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:10.976711 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:10.978172 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:10.978845 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:10.978848 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:10.978849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:10.978961 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:10.979461 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:10.979465 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:10.979466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:10.979758 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:10.979846 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:10.980531 (+ 685us) server_negotiation.cc:299] Negotiation successful
0925 04:39:10.980668 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260925 04:39:10.981742 4966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:10.968061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37853 (local address 127.0.98.3:33147)
0925 04:39:10.968647 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:10.968713 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:10.970086 (+ 1373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:10.970662 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:10.970672 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:10.971086 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:10.971648 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:10.971664 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:10.972502 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:10.972505 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:10.973449 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:10.973460 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:10.973969 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:10.973973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:10.974143 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:10.974756 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:10.974775 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:10.976480 (+ 1705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:39:10.978326 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:10.978332 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:10.978347 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:10.978651 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:10.979084 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:10.979088 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:10.979090 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:10.979337 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:10.979875 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:39:10.979883 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:39:10.980235 (+ 352us) client_negotiation.cc:769] Sending connection context
0925 04:39:10.980448 (+ 213us) client_negotiation.cc:240] Negotiation successful
0925 04:39:10.980815 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":144,"threads_started":1}
I20260925 04:39:10.982788 4964 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37853
I20260925 04:39:10.983021 4964 heartbeater.cc:464] Registering TS with master...
I20260925 04:39:10.983477 4964 heartbeater.cc:511] Master 127.0.98.62:37853 requested a full tablet report, sending...
I20260925 04:39:10.984572 4503 ts_manager.cc:194] Registered new tserver with Master: 4a8ce6d32ddc4a4aa99777adc2733cfa (127.0.98.3:38797)
I20260925 04:39:10.985153 4503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33147
I20260925 04:39:10.993391 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:39:11.002974 4547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:10.995339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41030 (local address 127.0.98.62:37853)
0925 04:39:10.995516 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:10.995520 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:10.995598 (+ 78us) server_negotiation.cc:407] Connection header received
0925 04:39:10.995736 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:10.995738 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:10.995783 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:10.995871 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:10.996583 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:10.997127 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:10.997985 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:10.998196 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:10.998272 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:10.999713 (+ 1441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:10.999718 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:10.999720 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:10.999748 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:11.001181 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:11.001656 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:11.001660 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:11.001661 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:11.001758 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:11.002034 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:11.002036 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:11.002037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:11.002269 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:11.002376 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:11.002660 (+ 284us) server_negotiation.cc:299] Negotiation successful
0925 04:39:11.002806 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 04:39:11.006358 4503 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41030:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:39:11.009649 4503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:39:11.026150 4980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:11.020685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47586 (local address 127.0.98.2:43171)
0925 04:39:11.020989 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:11.021000 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:11.021018 (+ 18us) server_negotiation.cc:407] Connection header received
0925 04:39:11.021083 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:11.021087 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:11.021341 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:11.021762 (+ 421us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:39:11.021914 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:11.022993 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:11.024498 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:11.025370 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:11.025482 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:11.025636 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:11.025861 (+ 225us) server_negotiation.cc:299] Negotiation successful
0925 04:39:11.025974 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":104,"threads_started":1}
I20260925 04:39:11.026149 4975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:11.019838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38797 (local address 127.0.0.1:49716)
0925 04:39:11.020378 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:11.020390 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:11.020453 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:11.021400 (+ 947us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:11.021403 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:11.021429 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:39:11.021584 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:11.021592 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:11.022716 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:11.022719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:11.024404 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:11.024415 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:11.025714 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:11.025718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:11.025797 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:11.025822 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 04:39:11.025890 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 04:39:11.025987 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"mutex_wait_us":174,"thread_start_us":97,"threads_started":1}
I20260925 04:39:11.026158 4978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:11.020269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43171 (local address 127.0.0.1:47586)
0925 04:39:11.020617 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:11.020628 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:11.020709 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:11.021490 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:11.021492 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:11.021509 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:39:11.021623 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:11.021629 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:11.023124 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:11.023128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:11.024371 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:11.024383 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:11.025498 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:11.025502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:11.025628 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:11.025671 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 04:39:11.025792 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 04:39:11.025973 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":42,"thread_start_us":89,"threads_started":1}
I20260925 04:39:11.026638 4979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:11.020521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49716 (local address 127.0.98.3:38797)
0925 04:39:11.020950 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:11.020961 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:11.020982 (+ 21us) server_negotiation.cc:407] Connection header received
0925 04:39:11.021052 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:11.021057 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:11.021255 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:11.021398 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:39:11.021707 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:11.022557 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:11.024526 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:11.025370 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:11.025482 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:11.025636 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:11.026373 (+ 737us) server_negotiation.cc:299] Negotiation successful
0925 04:39:11.026492 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":157,"threads_started":1}
I20260925 04:39:11.027060 4976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:11.019668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35670 (local address 127.0.98.1:44215)
0925 04:39:11.020668 (+ 1000us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:11.020676 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:11.020700 (+ 24us) server_negotiation.cc:407] Connection header received
0925 04:39:11.020776 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:11.020781 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:11.021013 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:11.021174 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:39:11.021502 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:11.022684 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:11.025187 (+ 2503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:11.025904 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:11.026609 (+ 705us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:11.026777 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:11.026828 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 04:39:11.026917 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":896,"thread_start_us":101,"threads_started":1}
I20260925 04:39:11.027316 4977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:11.019464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44215 (local address 127.0.0.1:35670)
0925 04:39:11.020203 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:11.020262 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:11.020396 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:11.021182 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:11.021188 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:11.021213 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:39:11.021360 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:11.021367 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:11.022822 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:11.022825 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:11.025052 (+ 2227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:11.025062 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:11.026025 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:11.026028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:11.026116 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:11.026136 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 04:39:11.026181 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 04:39:11.026242 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"mutex_wait_us":75,"thread_start_us":122,"threads_started":1}
I20260925 04:39:11.029534 4898 tablet_service.cc:1467] Processing CreateTablet for tablet 657dfc893c15427c923f6297239873c5 (DEFAULT_TABLE table=test-table [id=09a4e9a5c5ea4a7d9669ddeed7ba5074]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:39:11.029668 4765 tablet_service.cc:1467] Processing CreateTablet for tablet 657dfc893c15427c923f6297239873c5 (DEFAULT_TABLE table=test-table [id=09a4e9a5c5ea4a7d9669ddeed7ba5074]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:39:11.030040 4632 tablet_service.cc:1467] Processing CreateTablet for tablet 657dfc893c15427c923f6297239873c5 (DEFAULT_TABLE table=test-table [id=09a4e9a5c5ea4a7d9669ddeed7ba5074]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:39:11.030704 4898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 657dfc893c15427c923f6297239873c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:39:11.030761 4765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 657dfc893c15427c923f6297239873c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:39:11.030925 4632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 657dfc893c15427c923f6297239873c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:39:11.037551 4981 tablet_bootstrap.cc:492] T 657dfc893c15427c923f6297239873c5 P 4a8ce6d32ddc4a4aa99777adc2733cfa: Bootstrap starting.
I20260925 04:39:11.038744 4982 tablet_bootstrap.cc:492] T 657dfc893c15427c923f6297239873c5 P 681076b637134dc89507bd7b4da94664: Bootstrap starting.
I20260925 04:39:11.039049 4983 tablet_bootstrap.cc:492] T 657dfc893c15427c923f6297239873c5 P 89eaea2b54794ba59550154e057c4627: Bootstrap starting.
I20260925 04:39:11.039961 4981 tablet_bootstrap.cc:654] T 657dfc893c15427c923f6297239873c5 P 4a8ce6d32ddc4a4aa99777adc2733cfa: Neither blocks nor log segments found. Creating new log.
I20260925 04:39:11.041039 4981 log.cc:824] T 657dfc893c15427c923f6297239873c5 P 4a8ce6d32ddc4a4aa99777adc2733cfa: Log is configured to *not* fsync() on all Append() calls
I20260925 04:39:11.041139 4982 tablet_bootstrap.cc:654] T 657dfc893c15427c923f6297239873c5 P 681076b637134dc89507bd7b4da94664: Neither blocks nor log segments found. Creating new log.
I20260925 04:39:11.041138 4983 tablet_bootstrap.cc:654] T 657dfc893c15427c923f6297239873c5 P 89eaea2b54794ba59550154e057c4627: Neither blocks nor log segments found. Creating new log.
I20260925 04:39:11.042062 4982 log.cc:824] T 657dfc893c15427c923f6297239873c5 P 681076b637134dc89507bd7b4da94664: Log is configured to *not* fsync() on all Append() calls
I20260925 04:39:11.042264 4983 log.cc:824] T 657dfc893c15427c923f6297239873c5 P 89eaea2b54794ba59550154e057c4627: Log is configured to *not* fsync() on all Append() calls
I20260925 04:39:11.043310 4981 tablet_bootstrap.cc:492] T 657dfc893c15427c923f6297239873c5 P 4a8ce6d32ddc4a4aa99777adc2733cfa: No bootstrap required, opened a new log
I20260925 04:39:11.043515 4981 ts_tablet_manager.cc:1397] T 657dfc893c15427c923f6297239873c5 P 4a8ce6d32ddc4a4aa99777adc2733cfa: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 04:39:11.044382 4983 tablet_bootstrap.cc:492] T 657dfc893c15427c923f6297239873c5 P 89eaea2b54794ba59550154e057c4627: No bootstrap required, opened a new log
I20260925 04:39:11.044435 4982 tablet_bootstrap.cc:492] T 657dfc893c15427c923f6297239873c5 P 681076b637134dc89507bd7b4da94664: No bootstrap required, opened a new log
I20260925 04:39:11.044589 4982 ts_tablet_manager.cc:1397] T 657dfc893c15427c923f6297239873c5 P 681076b637134dc89507bd7b4da94664: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260925 04:39:11.044589 4983 ts_tablet_manager.cc:1397] T 657dfc893c15427c923f6297239873c5 P 89eaea2b54794ba59550154e057c4627: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 04:39:11.046736 4981 raft_consensus.cc:348] T 657dfc893c15427c923f6297239873c5 P 4a8ce6d32ddc4a4aa99777adc2733cfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681076b637134dc89507bd7b4da94664" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44215 } } peers { permanent_uuid: "89eaea2b54794ba59550154e057c4627" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43171 } } peers { permanent_uuid: "4a8ce6d32ddc4a4aa99777adc2733cfa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38797 } }
I20260925 04:39:11.047014 4981 raft_consensus.cc:374] T 657dfc893c15427c923f6297239873c5 P 4a8ce6d32ddc4a4aa99777adc2733cfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:39:11.047070 4981 raft_consensus.cc:729] T 657dfc893c15427c923f6297239873c5 P 4a8ce6d32ddc4a4aa99777adc2733cfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a8ce6d32ddc4a4aa99777adc2733cfa, State: Initialized, Role: FOLLOWER
I20260925 04:39:11.047549 4981 consensus_queue.cc:260] T 657dfc893c15427c923f6297239873c5 P 4a8ce6d32ddc4a4aa99777adc2733cfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681076b637134dc89507bd7b4da94664" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44215 } } peers { permanent_uuid: "89eaea2b54794ba59550154e057c4627" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43171 } } peers { permanent_uuid: "4a8ce6d32ddc4a4aa99777adc2733cfa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38797 } }
I20260925 04:39:11.048295 4964 heartbeater.cc:503] Master 127.0.98.62:37853 was elected leader, sending a full tablet report...
I20260925 04:39:11.048715 4981 ts_tablet_manager.cc:1428] T 657dfc893c15427c923f6297239873c5 P 4a8ce6d32ddc4a4aa99777adc2733cfa: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 04:39:11.048720 4983 raft_consensus.cc:348] T 657dfc893c15427c923f6297239873c5 P 89eaea2b54794ba59550154e057c4627 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681076b637134dc89507bd7b4da94664" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44215 } } peers { permanent_uuid: "89eaea2b54794ba59550154e057c4627" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43171 } } peers { permanent_uuid: "4a8ce6d32ddc4a4aa99777adc2733cfa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38797 } }
I20260925 04:39:11.049036 4983 raft_consensus.cc:374] T 657dfc893c15427c923f6297239873c5 P 89eaea2b54794ba59550154e057c4627 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:39:11.049093 4983 raft_consensus.cc:729] T 657dfc893c15427c923f6297239873c5 P 89eaea2b54794ba59550154e057c4627 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89eaea2b54794ba59550154e057c4627, State: Initialized, Role: FOLLOWER
I20260925 04:39:11.048997 4982 raft_consensus.cc:348] T 657dfc893c15427c923f6297239873c5 P 681076b637134dc89507bd7b4da94664 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681076b637134dc89507bd7b4da94664" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44215 } } peers { permanent_uuid: "89eaea2b54794ba59550154e057c4627" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43171 } } peers { permanent_uuid: "4a8ce6d32ddc4a4aa99777adc2733cfa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38797 } }
I20260925 04:39:11.049243 4982 raft_consensus.cc:374] T 657dfc893c15427c923f6297239873c5 P 681076b637134dc89507bd7b4da94664 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:39:11.049301 4982 raft_consensus.cc:729] T 657dfc893c15427c923f6297239873c5 P 681076b637134dc89507bd7b4da94664 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 681076b637134dc89507bd7b4da94664, State: Initialized, Role: FOLLOWER
I20260925 04:39:11.049752 4982 consensus_queue.cc:260] T 657dfc893c15427c923f6297239873c5 P 681076b637134dc89507bd7b4da94664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681076b637134dc89507bd7b4da94664" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44215 } } peers { permanent_uuid: "89eaea2b54794ba59550154e057c4627" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43171 } } peers { permanent_uuid: "4a8ce6d32ddc4a4aa99777adc2733cfa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38797 } }
I20260925 04:39:11.050477 4698 heartbeater.cc:503] Master 127.0.98.62:37853 was elected leader, sending a full tablet report...
I20260925 04:39:11.050894 4982 ts_tablet_manager.cc:1428] T 657dfc893c15427c923f6297239873c5 P 681076b637134dc89507bd7b4da94664: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 04:39:11.049619 4983 consensus_queue.cc:260] T 657dfc893c15427c923f6297239873c5 P 89eaea2b54794ba59550154e057c4627 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681076b637134dc89507bd7b4da94664" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44215 } } peers { permanent_uuid: "89eaea2b54794ba59550154e057c4627" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43171 } } peers { permanent_uuid: "4a8ce6d32ddc4a4aa99777adc2733cfa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38797 } }
I20260925 04:39:11.052048 4831 heartbeater.cc:503] Master 127.0.98.62:37853 was elected leader, sending a full tablet report...
I20260925 04:39:11.052232 4983 ts_tablet_manager.cc:1428] T 657dfc893c15427c923f6297239873c5 P 89eaea2b54794ba59550154e057c4627: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260925 04:39:11.069854 4987 raft_consensus.cc:482] T 657dfc893c15427c923f6297239873c5 P 4a8ce6d32ddc4a4aa99777adc2733cfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:39:11.070047 4987 raft_consensus.cc:504] T 657dfc893c15427c923f6297239873c5 P 4a8ce6d32ddc4a4aa99777adc2733cfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681076b637134dc89507bd7b4da94664" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44215 } } peers { permanent_uuid: "89eaea2b54794ba59550154e057c4627" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43171 } } peers { permanent_uuid: "4a8ce6d32ddc4a4aa99777adc2733cfa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38797 } }
I20260925 04:39:11.071127 4987 leader_election.cc:288] T 657dfc893c15427c923f6297239873c5 P 4a8ce6d32ddc4a4aa99777adc2733cfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 681076b637134dc89507bd7b4da94664 (127.0.98.1:44215), 89eaea2b54794ba59550154e057c4627 (127.0.98.2:43171)
I20260925 04:39:11.075464 4966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:11.071417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44215 (local address 127.0.98.3:48813)
0925 04:39:11.071550 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:11.071566 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:11.071681 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:11.072056 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:11.072059 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:11.072080 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:39:11.072212 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:11.072220 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:11.073115 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:11.073119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:11.074084 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:11.074095 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:11.074980 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:11.074984 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:11.075093 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:11.075120 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 04:39:11.075245 (+ 125us) client_negotiation.cc:240] Negotiation successful
0925 04:39:11.075319 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 04:39:11.075492 4976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:11.071526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48813 (local address 127.0.98.1:44215)
0925 04:39:11.071669 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:11.071672 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:11.071686 (+ 14us) server_negotiation.cc:407] Connection header received
0925 04:39:11.071811 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:11.071815 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:11.071867 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:11.071953 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:39:11.072342 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:11.072987 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:11.074243 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:11.074852 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:11.074944 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:11.075003 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:11.075230 (+ 227us) server_negotiation.cc:299] Negotiation successful
0925 04:39:11.075362 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 04:39:11.075558 4980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:11.071907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52175 (local address 127.0.98.2:43171)
0925 04:39:11.072031 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:11.072035 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:11.072185 (+ 150us) server_negotiation.cc:407] Connection header received
0925 04:39:11.072343 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:11.072346 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:11.072390 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:11.072447 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:39:11.072748 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:11.073320 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:11.074480 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:11.075022 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:11.075125 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:11.075184 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:11.075384 (+ 200us) server_negotiation.cc:299] Negotiation successful
0925 04:39:11.075454 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 04:39:11.075623 4990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:11.071705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43171 (local address 127.0.98.3:52175)
0925 04:39:11.072088 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:11.072099 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:11.072196 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:11.072476 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:11.072479 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:11.072497 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:39:11.072610 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:11.072616 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:11.073461 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:11.073464 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:11.074346 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:11.074357 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:11.075145 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:11.075148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:11.075257 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:11.075279 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 04:39:11.075353 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 04:39:11.075419 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":141,"threads_started":1}
I20260925 04:39:11.076256 4785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "657dfc893c15427c923f6297239873c5" candidate_uuid: "4a8ce6d32ddc4a4aa99777adc2733cfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89eaea2b54794ba59550154e057c4627" is_pre_election: true
I20260925 04:39:11.076262 4652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "657dfc893c15427c923f6297239873c5" candidate_uuid: "4a8ce6d32ddc4a4aa99777adc2733cfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "681076b637134dc89507bd7b4da94664" is_pre_election: true
I20260925 04:39:11.076561 4652 raft_consensus.cc:2454] T 657dfc893c15427c923f6297239873c5 P 681076b637134dc89507bd7b4da94664 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a8ce6d32ddc4a4aa99777adc2733cfa in term 0.
I20260925 04:39:11.076562 4785 raft_consensus.cc:2454] T 657dfc893c15427c923f6297239873c5 P 89eaea2b54794ba59550154e057c4627 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a8ce6d32ddc4a4aa99777adc2733cfa in term 0.
I20260925 04:39:11.077068 4853 leader_election.cc:302] T 657dfc893c15427c923f6297239873c5 P 4a8ce6d32ddc4a4aa99777adc2733cfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a8ce6d32ddc4a4aa99777adc2733cfa, 89eaea2b54794ba59550154e057c4627; no voters:
I20260925 04:39:11.077376 4987 raft_consensus.cc:2781] T 657dfc893c15427c923f6297239873c5 P 4a8ce6d32ddc4a4aa99777adc2733cfa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:39:11.077468 4987 raft_consensus.cc:482] T 657dfc893c15427c923f6297239873c5 P 4a8ce6d32ddc4a4aa99777adc2733cfa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:39:11.077508 4987 raft_consensus.cc:3037] T 657dfc893c15427c923f6297239873c5 P 4a8ce6d32ddc4a4aa99777adc2733cfa [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:39:11.078565 4987 raft_consensus.cc:504] T 657dfc893c15427c923f6297239873c5 P 4a8ce6d32ddc4a4aa99777adc2733cfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681076b637134dc89507bd7b4da94664" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44215 } } peers { permanent_uuid: "89eaea2b54794ba59550154e057c4627" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43171 } } peers { permanent_uuid: "4a8ce6d32ddc4a4aa99777adc2733cfa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38797 } }
I20260925 04:39:11.078933 4987 leader_election.cc:288] T 657dfc893c15427c923f6297239873c5 P 4a8ce6d32ddc4a4aa99777adc2733cfa [CANDIDATE]: Term 1 election: Requested vote from peers 681076b637134dc89507bd7b4da94664 (127.0.98.1:44215), 89eaea2b54794ba59550154e057c4627 (127.0.98.2:43171)
I20260925 04:39:11.079367 4785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "657dfc893c15427c923f6297239873c5" candidate_uuid: "4a8ce6d32ddc4a4aa99777adc2733cfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89eaea2b54794ba59550154e057c4627"
I20260925 04:39:11.079368 4652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "657dfc893c15427c923f6297239873c5" candidate_uuid: "4a8ce6d32ddc4a4aa99777adc2733cfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "681076b637134dc89507bd7b4da94664"
I20260925 04:39:11.079509 4652 raft_consensus.cc:3037] T 657dfc893c15427c923f6297239873c5 P 681076b637134dc89507bd7b4da94664 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:39:11.079509 4785 raft_consensus.cc:3037] T 657dfc893c15427c923f6297239873c5 P 89eaea2b54794ba59550154e057c4627 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:39:11.080834 4785 raft_consensus.cc:2454] T 657dfc893c15427c923f6297239873c5 P 89eaea2b54794ba59550154e057c4627 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a8ce6d32ddc4a4aa99777adc2733cfa in term 1.
I20260925 04:39:11.080832 4652 raft_consensus.cc:2454] T 657dfc893c15427c923f6297239873c5 P 681076b637134dc89507bd7b4da94664 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a8ce6d32ddc4a4aa99777adc2733cfa in term 1.
I20260925 04:39:11.081199 4853 leader_election.cc:302] T 657dfc893c15427c923f6297239873c5 P 4a8ce6d32ddc4a4aa99777adc2733cfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a8ce6d32ddc4a4aa99777adc2733cfa, 89eaea2b54794ba59550154e057c4627; no voters:
I20260925 04:39:11.081422 4987 raft_consensus.cc:2781] T 657dfc893c15427c923f6297239873c5 P 4a8ce6d32ddc4a4aa99777adc2733cfa [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:39:11.081697 4987 raft_consensus.cc:686] T 657dfc893c15427c923f6297239873c5 P 4a8ce6d32ddc4a4aa99777adc2733cfa [term 1 LEADER]: Becoming Leader. State: Replica: 4a8ce6d32ddc4a4aa99777adc2733cfa, State: Running, Role: LEADER
I20260925 04:39:11.082120 4987 consensus_queue.cc:237] T 657dfc893c15427c923f6297239873c5 P 4a8ce6d32ddc4a4aa99777adc2733cfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681076b637134dc89507bd7b4da94664" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44215 } } peers { permanent_uuid: "89eaea2b54794ba59550154e057c4627" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43171 } } peers { permanent_uuid: "4a8ce6d32ddc4a4aa99777adc2733cfa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38797 } }
I20260925 04:39:11.085248 4507 catalog_manager.cc:5495] T 657dfc893c15427c923f6297239873c5 P 4a8ce6d32ddc4a4aa99777adc2733cfa reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a8ce6d32ddc4a4aa99777adc2733cfa (127.0.98.3). New cstate: current_term: 1 leader_uuid: "4a8ce6d32ddc4a4aa99777adc2733cfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681076b637134dc89507bd7b4da94664" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89eaea2b54794ba59550154e057c4627" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a8ce6d32ddc4a4aa99777adc2733cfa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38797 } health_report { overall_health: HEALTHY } } }
I20260925 04:39:11.107530 4979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:11.103465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49730 (local address 127.0.98.3:38797)
0925 04:39:11.103626 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:11.103630 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:11.103642 (+ 12us) server_negotiation.cc:407] Connection header received
0925 04:39:11.103732 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:11.103734 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:11.103791 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:11.103886 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:39:11.104139 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:11.104731 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:11.105912 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:11.106132 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:11.106222 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:11.106558 (+ 336us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:39:11.107006 (+ 448us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:39:11.107117 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:11.107251 (+ 134us) server_negotiation.cc:299] Negotiation successful
0925 04:39:11.107313 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 04:39:11.113974 4785 raft_consensus.cc:1261] T 657dfc893c15427c923f6297239873c5 P 89eaea2b54794ba59550154e057c4627 [term 1 FOLLOWER]: Refusing update from remote peer 4a8ce6d32ddc4a4aa99777adc2733cfa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:39:11.113976 4652 raft_consensus.cc:1261] T 657dfc893c15427c923f6297239873c5 P 681076b637134dc89507bd7b4da94664 [term 1 FOLLOWER]: Refusing update from remote peer 4a8ce6d32ddc4a4aa99777adc2733cfa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:39:11.114733 4991 consensus_queue.cc:1035] T 657dfc893c15427c923f6297239873c5 P 4a8ce6d32ddc4a4aa99777adc2733cfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "681076b637134dc89507bd7b4da94664" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:39:11.115047 4987 consensus_queue.cc:1035] T 657dfc893c15427c923f6297239873c5 P 4a8ce6d32ddc4a4aa99777adc2733cfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "89eaea2b54794ba59550154e057c4627" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:39:11.124934 4996 mvcc.cc:205] Tried to move back new op lower bound from 7333114474955046912 to 7333114474835324928. Current Snapshot: MvccSnapshot[applied={T|T < 7333114474955046912}]
I20260925 04:39:11.132463 4508 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41030:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:39:11.132642 4508 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41030:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:39:11.135587 4508 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 06cd63cb831f47ceab62dc54f45ef0c7: Sending DeleteTablet for 3 replicas of tablet 657dfc893c15427c923f6297239873c5
I20260925 04:39:11.136689 4765 tablet_service.cc:1514] Processing DeleteTablet for tablet 657dfc893c15427c923f6297239873c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:39:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47586
I20260925 04:39:11.136804 4632 tablet_service.cc:1514] Processing DeleteTablet for tablet 657dfc893c15427c923f6297239873c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:39:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35670
I20260925 04:39:11.136977 4898 tablet_service.cc:1514] Processing DeleteTablet for tablet 657dfc893c15427c923f6297239873c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:39:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49716
I20260925 04:39:11.137470 5004 tablet_replica.cc:330] T 657dfc893c15427c923f6297239873c5 P 89eaea2b54794ba59550154e057c4627: stopping tablet replica
I20260925 04:39:11.137600 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 4570
I20260925 04:39:11.137854 5004 raft_consensus.cc:2229] T 657dfc893c15427c923f6297239873c5 P 89eaea2b54794ba59550154e057c4627 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:39:11.138230 5004 raft_consensus.cc:2258] T 657dfc893c15427c923f6297239873c5 P 89eaea2b54794ba59550154e057c4627 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:39:11.138665 5006 tablet_replica.cc:330] T 657dfc893c15427c923f6297239873c5 P 4a8ce6d32ddc4a4aa99777adc2733cfa: stopping tablet replica
I20260925 04:39:11.138983 5006 raft_consensus.cc:2229] T 657dfc893c15427c923f6297239873c5 P 4a8ce6d32ddc4a4aa99777adc2733cfa [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:39:11.139865 5006 raft_consensus.cc:2258] T 657dfc893c15427c923f6297239873c5 P 4a8ce6d32ddc4a4aa99777adc2733cfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:39:11.141206 5004 ts_tablet_manager.cc:1905] T 657dfc893c15427c923f6297239873c5 P 89eaea2b54794ba59550154e057c4627: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:39:11.141351 5006 ts_tablet_manager.cc:1905] T 657dfc893c15427c923f6297239873c5 P 4a8ce6d32ddc4a4aa99777adc2733cfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:39:11.144447 5006 ts_tablet_manager.cc:1918] T 657dfc893c15427c923f6297239873c5 P 4a8ce6d32ddc4a4aa99777adc2733cfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:39:11.144575 5006 log.cc:1196] T 657dfc893c15427c923f6297239873c5 P 4a8ce6d32ddc4a4aa99777adc2733cfa: Deleting WAL directory at /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/minicluster-data/ts-2/wal/wals/657dfc893c15427c923f6297239873c5
I20260925 04:39:11.144677 5004 ts_tablet_manager.cc:1918] T 657dfc893c15427c923f6297239873c5 P 89eaea2b54794ba59550154e057c4627: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:39:11.144855 5004 log.cc:1196] T 657dfc893c15427c923f6297239873c5 P 89eaea2b54794ba59550154e057c4627: Deleting WAL directory at /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311123478858-392-0/minicluster-data/ts-1/wal/wals/657dfc893c15427c923f6297239873c5
I20260925 04:39:11.144917 5006 ts_tablet_manager.cc:1939] T 657dfc893c15427c923f6297239873c5 P 4a8ce6d32ddc4a4aa99777adc2733cfa: Deleting consensus metadata
I20260925 04:39:11.145256 5004 ts_tablet_manager.cc:1939] T 657dfc893c15427c923f6297239873c5 P 89eaea2b54794ba59550154e057c4627: Deleting consensus metadata
W20260925 04:39:11.145752 4494 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44215 (error 108)
I20260925 04:39:11.146261 4495 catalog_manager.cc:4841] TS 4a8ce6d32ddc4a4aa99777adc2733cfa (127.0.98.3:38797): tablet 657dfc893c15427c923f6297239873c5 (table test-table [id=09a4e9a5c5ea4a7d9669ddeed7ba5074]) successfully deleted
I20260925 04:39:11.146595 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 4703
I20260925 04:39:11.146695 4494 catalog_manager.cc:4841] TS 89eaea2b54794ba59550154e057c4627 (127.0.98.2:43171): tablet 657dfc893c15427c923f6297239873c5 (table test-table [id=09a4e9a5c5ea4a7d9669ddeed7ba5074]) successfully deleted
I20260925 04:39:11.148080 4977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:11.147716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44215 (local address 127.0.0.1:35682)
0925 04:39:11.147896 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:11.147994 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260925 04:39:11.148337 4494 catalog_manager.cc:4584] TS 681076b637134dc89507bd7b4da94664 (127.0.98.1:44215): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 657dfc893c15427c923f6297239873c5: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44215: connect: Connection refused (error 111)
I20260925 04:39:11.152105 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 4836
I20260925 04:39:11.157369 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 4477
2026-09-25T04:39:11Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2273 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-25T04:39:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:39:11Z Disabled control of system clock
I20260925 04:39:11.186897 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311123478858-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311123478858-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311123478858-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33223
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33223
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 04:39:11.301537 5011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:39:11.301839 5011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:39:11.301878 5011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:39:11.306087 5011 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:39:11.306169 5011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:39:11.306190 5011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:39:11.306209 5011 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:39:11.306224 5011 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:39:11.311204 5011 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311123478858-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311123478858-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33223
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311123478858-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:39:11.312477 5011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:39:11.313935 5011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:39:11.322594 5017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:11.322585 5016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:11.322580 5019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:11.322887 5011 server_base.cc:1034] running on GCE node
I20260925 04:39:11.323659 5011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:39:11.324908 5011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:39:11.326159 5011 hybrid_clock.cc:648] HybridClock initialized: now 1790311151326133 us; error 41 us; skew 500 ppm
I20260925 04:39:11.334760 5011 webserver.cc:460] Webserver started at http://127.0.98.62:34599/ using document root <none> and password file <none>
I20260925 04:39:11.335490 5011 fs_manager.cc:359] Metadata directory not provided
I20260925 04:39:11.335582 5011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:39:11.335852 5011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:39:11.338133 5011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311123478858-392-0/minicluster-data/master-0/data/instance:
uuid: "53abdf3881c649e0b30cd244403f464f"
format_stamp: "Formatted at 2026-09-25 04:39:11 on dist-test-slave-dc95"
server_key: "59d3bcf28419d40dc203bcda3df50e37"
server_key_iv: "21e67119c6b58d16ba76c640b0eaf1c0"
server_key_version: "encryptionkey@0"
I20260925 04:39:11.338743 5011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311123478858-392-0/minicluster-data/master-0/wal/instance:
uuid: "53abdf3881c649e0b30cd244403f464f"
format_stamp: "Formatted at 2026-09-25 04:39:11 on dist-test-slave-dc95"
server_key: "59d3bcf28419d40dc203bcda3df50e37"
server_key_iv: "21e67119c6b58d16ba76c640b0eaf1c0"
server_key_version: "encryptionkey@0"
I20260925 04:39:11.342911 5011 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 04:39:11.345827 5025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:11.347462 5011 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:39:11.347649 5011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311123478858-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311123478858-392-0/minicluster-data/master-0/wal
uuid: "53abdf3881c649e0b30cd244403f464f"
format_stamp: "Formatted at 2026-09-25 04:39:11 on dist-test-slave-dc95"
server_key: "59d3bcf28419d40dc203bcda3df50e37"
server_key_iv: "21e67119c6b58d16ba76c640b0eaf1c0"
server_key_version: "encryptionkey@0"
I20260925 04:39:11.347783 5011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311123478858-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311123478858-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311123478858-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:39:11.352967 5011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:39:11.353725 5011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:39:11.353914 5011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:39:11.361142 5011 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33223
I20260925 04:39:11.361151 5077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33223 every 8 connection(s)
I20260925 04:39:11.362195 5011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
I20260925 04:39:11.363394 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 5011
I20260925 04:39:11.363511 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311123478858-392-0/minicluster-data/master-0/wal/instance
I20260925 04:39:11.363897 392 external_mini_cluster.cc:1428] Setting key 73f996d8ae33fe27e82996f017df241d
I20260925 04:39:11.366128 5078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:39:11.371279 5081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:11.365439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48488 (local address 127.0.98.62:33223)
0925 04:39:11.366847 (+ 1408us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:11.366866 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:11.366909 (+ 43us) server_negotiation.cc:407] Connection header received
0925 04:39:11.367581 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:11.367599 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:11.368307 (+ 708us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:11.368975 (+ 668us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:11.369894 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:11.369899 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:11.369916 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:39:11.369926 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:11.369987 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:39:11.370005 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:11.370152 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:11.370228 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 04:39:11.370497 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":982,"thread_start_us":711,"threads_started":1}
I20260925 04:39:11.372879 5078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53abdf3881c649e0b30cd244403f464f: Bootstrap starting.
I20260925 04:39:11.375532 5078 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53abdf3881c649e0b30cd244403f464f: Neither blocks nor log segments found. Creating new log.
I20260925 04:39:11.376533 5078 log.cc:824] T 00000000000000000000000000000000 P 53abdf3881c649e0b30cd244403f464f: Log is configured to *not* fsync() on all Append() calls
I20260925 04:39:11.379856 5078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53abdf3881c649e0b30cd244403f464f: No bootstrap required, opened a new log
I20260925 04:39:11.383571 5078 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53abdf3881c649e0b30cd244403f464f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53abdf3881c649e0b30cd244403f464f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33223 } }
I20260925 04:39:11.383908 5078 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53abdf3881c649e0b30cd244403f464f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:39:11.383982 5078 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53abdf3881c649e0b30cd244403f464f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53abdf3881c649e0b30cd244403f464f, State: Initialized, Role: FOLLOWER
I20260925 04:39:11.384454 5078 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53abdf3881c649e0b30cd244403f464f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53abdf3881c649e0b30cd244403f464f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33223 } }
I20260925 04:39:11.384600 5078 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53abdf3881c649e0b30cd244403f464f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:39:11.384670 5078 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53abdf3881c649e0b30cd244403f464f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:39:11.384773 5078 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53abdf3881c649e0b30cd244403f464f [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:39:11.385861 5078 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53abdf3881c649e0b30cd244403f464f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53abdf3881c649e0b30cd244403f464f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33223 } }
I20260925 04:39:11.386214 5078 leader_election.cc:302] T 00000000000000000000000000000000 P 53abdf3881c649e0b30cd244403f464f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53abdf3881c649e0b30cd244403f464f; no voters:
I20260925 04:39:11.386514 5078 leader_election.cc:288] T 00000000000000000000000000000000 P 53abdf3881c649e0b30cd244403f464f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:39:11.386650 5083 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53abdf3881c649e0b30cd244403f464f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:39:11.386893 5083 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53abdf3881c649e0b30cd244403f464f [term 1 LEADER]: Becoming Leader. State: Replica: 53abdf3881c649e0b30cd244403f464f, State: Running, Role: LEADER
I20260925 04:39:11.387212 5083 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53abdf3881c649e0b30cd244403f464f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53abdf3881c649e0b30cd244403f464f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33223 } }
I20260925 04:39:11.387821 5078 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53abdf3881c649e0b30cd244403f464f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:39:11.388979 5084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53abdf3881c649e0b30cd244403f464f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53abdf3881c649e0b30cd244403f464f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53abdf3881c649e0b30cd244403f464f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33223 } } }
I20260925 04:39:11.389110 5084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53abdf3881c649e0b30cd244403f464f [sys.catalog]: This master's current role is: LEADER
I20260925 04:39:11.389497 5092 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:39:11.389438 5085 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53abdf3881c649e0b30cd244403f464f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53abdf3881c649e0b30cd244403f464f. Latest consensus state: current_term: 1 leader_uuid: "53abdf3881c649e0b30cd244403f464f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53abdf3881c649e0b30cd244403f464f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33223 } } }
I20260925 04:39:11.389603 5085 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53abdf3881c649e0b30cd244403f464f [sys.catalog]: This master's current role is: LEADER
I20260925 04:39:11.393644 5092 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:39:11.400471 5092 catalog_manager.cc:1347] Generated new cluster ID: 022aaef3344e42fc8667c0443e793c59
I20260925 04:39:11.400574 5092 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:39:11.411376 5092 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:39:11.411830 5092 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:39:11.421034 5092 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 53abdf3881c649e0b30cd244403f464f: Generated new TSK 0
I20260925 04:39:11.421933 5092 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:39:11.431988 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311123478858-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311123478858-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311123478858-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33223
--builtin_ntp_servers=127.0.98.20:45293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 04:39:11.553010 5102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:39:11.553393 5102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:39:11.553473 5102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:39:11.557838 5102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:39:11.557914 5102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:39:11.558001 5102 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 04:39:11.563112 5102 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311123478858-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311123478858-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311123478858-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:39:11.564357 5102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:39:11.565822 5102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:39:11.576323 5110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:11.576381 5108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:11.576427 5102 server_base.cc:1034] running on GCE node
W20260925 04:39:11.576344 5107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:11.576942 5102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:39:11.577649 5102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:39:11.578840 5102 hybrid_clock.cc:648] HybridClock initialized: now 1790311151578804 us; error 45 us; skew 500 ppm
I20260925 04:39:11.581008 5102 webserver.cc:460] Webserver started at http://127.0.98.1:44241/ using document root <none> and password file <none>
I20260925 04:39:11.581619 5102 fs_manager.cc:359] Metadata directory not provided
I20260925 04:39:11.581689 5102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:39:11.582002 5102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:39:11.584261 5102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311123478858-392-0/minicluster-data/ts-0/data/instance:
uuid: "0d273bd980ce43c88852bb7e1db08425"
format_stamp: "Formatted at 2026-09-25 04:39:11 on dist-test-slave-dc95"
server_key: "4e1d98847556a95465ecee0e071b69c0"
server_key_iv: "14dbbdfa3c2bb6f884bd4957bc88ffa4"
server_key_version: "encryptionkey@0"
I20260925 04:39:11.584831 5102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311123478858-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0d273bd980ce43c88852bb7e1db08425"
format_stamp: "Formatted at 2026-09-25 04:39:11 on dist-test-slave-dc95"
server_key: "4e1d98847556a95465ecee0e071b69c0"
server_key_iv: "14dbbdfa3c2bb6f884bd4957bc88ffa4"
server_key_version: "encryptionkey@0"
I20260925 04:39:11.588884 5102 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260925 04:39:11.591739 5116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:11.593221 5102 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:39:11.593349 5102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311123478858-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311123478858-392-0/minicluster-data/ts-0/wal
uuid: "0d273bd980ce43c88852bb7e1db08425"
format_stamp: "Formatted at 2026-09-25 04:39:11 on dist-test-slave-dc95"
server_key: "4e1d98847556a95465ecee0e071b69c0"
server_key_iv: "14dbbdfa3c2bb6f884bd4957bc88ffa4"
server_key_version: "encryptionkey@0"
I20260925 04:39:11.593485 5102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311123478858-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311123478858-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311123478858-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:39:11.598382 5102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:39:11.599058 5102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:39:11.599228 5102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:39:11.599841 5102 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:39:11.600823 5102 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:39:11.600879 5102 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:11.600930 5102 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:39:11.600958 5102 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:11.610227 5102 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45451
I20260925 04:39:11.610257 5229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45451 every 8 connection(s)
I20260925 04:39:11.611212 5102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
I20260925 04:39:11.617681 5081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:11.613280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53639 (local address 127.0.98.62:33223)
0925 04:39:11.613432 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:11.613435 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:11.614656 (+ 1221us) server_negotiation.cc:407] Connection header received
0925 04:39:11.616013 (+ 1357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:11.616016 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:11.616056 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:11.616118 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:11.617237 (+ 1119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:11.617240 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:11.617242 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:39:11.617253 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:11.617273 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:39:11.617281 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:11.617367 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:11.617538 (+ 171us) server_negotiation.cc:299] Negotiation successful
0925 04:39:11.617573 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 04:39:11.618564 5232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:11.613696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33223 (local address 127.0.98.1:53639)
0925 04:39:11.614465 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:11.614508 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:11.615795 (+ 1287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:11.616313 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:11.616320 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:11.616414 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:11.616936 (+ 522us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:39:11.616938 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:11.616980 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:39:11.617050 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:39:11.617405 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:39:11.617413 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:39:11.617418 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 04:39:11.617573 (+ 155us) client_negotiation.cc:240] Negotiation successful
0925 04:39:11.617776 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":120,"threads_started":1}
I20260925 04:39:11.618950 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 5102
I20260925 04:39:11.619058 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311123478858-392-0/minicluster-data/ts-0/wal/instance
I20260925 04:39:11.619385 392 external_mini_cluster.cc:1428] Setting key 6437b2ae5f7c837e4fc6c4242d3143ea
I20260925 04:39:11.619880 5230 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33223
I20260925 04:39:11.620123 5230 heartbeater.cc:464] Registering TS with master...
I20260925 04:39:11.620554 5230 heartbeater.cc:511] Master 127.0.98.62:33223 requested a full tablet report, sending...
I20260925 04:39:11.621315 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311123478858-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311123478858-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311123478858-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33223
--builtin_ntp_servers=127.0.98.20:45293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 04:39:11.622203 5042 ts_manager.cc:194] Registered new tserver with Master: 0d273bd980ce43c88852bb7e1db08425 (127.0.98.1:45451)
W20260925 04:39:11.736560 5233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:39:11.736857 5233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:39:11.736899 5233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:39:11.741539 5233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:39:11.741631 5233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:39:11.741693 5233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 04:39:11.746721 5233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311123478858-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311123478858-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311123478858-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:39:11.747922 5233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:39:11.749318 5233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:39:11.759791 5239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:11.759791 5238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:11.759817 5241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:11.760195 5233 server_base.cc:1034] running on GCE node
I20260925 04:39:11.760614 5233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:39:11.761300 5233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:39:11.762487 5233 hybrid_clock.cc:648] HybridClock initialized: now 1790311151762468 us; error 34 us; skew 500 ppm
I20260925 04:39:11.764577 5233 webserver.cc:460] Webserver started at http://127.0.98.2:37257/ using document root <none> and password file <none>
I20260925 04:39:11.765174 5233 fs_manager.cc:359] Metadata directory not provided
I20260925 04:39:11.765242 5233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:39:11.765461 5233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:39:11.767776 5233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311123478858-392-0/minicluster-data/ts-1/data/instance:
uuid: "987b0a35fc65487ea8cbad5abda7d4e0"
format_stamp: "Formatted at 2026-09-25 04:39:11 on dist-test-slave-dc95"
server_key: "2a28d66bef24a5977559bfca026daaa8"
server_key_iv: "c94625ee6d0e743eb335378760aa1f3f"
server_key_version: "encryptionkey@0"
I20260925 04:39:11.768451 5233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311123478858-392-0/minicluster-data/ts-1/wal/instance:
uuid: "987b0a35fc65487ea8cbad5abda7d4e0"
format_stamp: "Formatted at 2026-09-25 04:39:11 on dist-test-slave-dc95"
server_key: "2a28d66bef24a5977559bfca026daaa8"
server_key_iv: "c94625ee6d0e743eb335378760aa1f3f"
server_key_version: "encryptionkey@0"
I20260925 04:39:11.772740 5233 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:39:11.775557 5247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:11.777016 5233 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:39:11.777149 5233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311123478858-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311123478858-392-0/minicluster-data/ts-1/wal
uuid: "987b0a35fc65487ea8cbad5abda7d4e0"
format_stamp: "Formatted at 2026-09-25 04:39:11 on dist-test-slave-dc95"
server_key: "2a28d66bef24a5977559bfca026daaa8"
server_key_iv: "c94625ee6d0e743eb335378760aa1f3f"
server_key_version: "encryptionkey@0"
I20260925 04:39:11.777268 5233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311123478858-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311123478858-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311123478858-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:39:11.782394 5233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:39:11.783115 5233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:39:11.783296 5233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:39:11.783950 5233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:39:11.784907 5233 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:39:11.784955 5233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:11.785007 5233 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:39:11.785034 5233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:11.793721 5233 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45379
I20260925 04:39:11.793781 5360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45379 every 8 connection(s)
I20260925 04:39:11.794665 5233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
I20260925 04:39:11.798790 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 5233
I20260925 04:39:11.798921 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311123478858-392-0/minicluster-data/ts-1/wal/instance
I20260925 04:39:11.799234 392 external_mini_cluster.cc:1428] Setting key 0002fc41c50e8fbd5f7395e028478082
I20260925 04:39:11.801493 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311123478858-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311123478858-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311123478858-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33223
--builtin_ntp_servers=127.0.98.20:45293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 04:39:11.801812 5081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:11.796778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53043 (local address 127.0.98.62:33223)
0925 04:39:11.796940 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:11.796944 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:11.798273 (+ 1329us) server_negotiation.cc:407] Connection header received
0925 04:39:11.799964 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:11.799968 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:11.800023 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:11.800112 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:11.801337 (+ 1225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:11.801340 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:11.801341 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:39:11.801354 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:11.801375 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:39:11.801383 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:11.801473 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:11.801640 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 04:39:11.801690 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 04:39:11.802827 5363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:11.797293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33223 (local address 127.0.98.2:53043)
0925 04:39:11.798091 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:11.798138 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:11.799664 (+ 1526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:11.800327 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:11.800336 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:11.800448 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:11.801035 (+ 587us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:39:11.801037 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:11.801088 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:39:11.801166 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:39:11.801462 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:39:11.801470 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:39:11.801476 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 04:39:11.801668 (+ 192us) client_negotiation.cc:240] Negotiation successful
0925 04:39:11.801983 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":138,"threads_started":1}
I20260925 04:39:11.804081 5361 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33223
I20260925 04:39:11.804332 5361 heartbeater.cc:464] Registering TS with master...
I20260925 04:39:11.804777 5361 heartbeater.cc:511] Master 127.0.98.62:33223 requested a full tablet report, sending...
I20260925 04:39:11.805931 5042 ts_manager.cc:194] Registered new tserver with Master: 987b0a35fc65487ea8cbad5abda7d4e0 (127.0.98.2:45379)
W20260925 04:39:11.919788 5364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:39:11.920076 5364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:39:11.920120 5364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:39:11.924743 5364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:39:11.924822 5364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:39:11.924888 5364 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 04:39:11.930138 5364 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311123478858-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311123478858-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5364
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311123478858-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:39:11.931313 5364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:39:11.932883 5364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:39:11.943024 5372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:11.943041 5369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:11.943041 5370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:11.943315 5364 server_base.cc:1034] running on GCE node
I20260925 04:39:11.943926 5364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:39:11.944594 5364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:39:11.945770 5364 hybrid_clock.cc:648] HybridClock initialized: now 1790311151945738 us; error 63 us; skew 500 ppm
I20260925 04:39:11.947923 5364 webserver.cc:460] Webserver started at http://127.0.98.3:45827/ using document root <none> and password file <none>
I20260925 04:39:11.948534 5364 fs_manager.cc:359] Metadata directory not provided
I20260925 04:39:11.948601 5364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:39:11.948791 5364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:39:11.951002 5364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311123478858-392-0/minicluster-data/ts-2/data/instance:
uuid: "76b56ce822c34819aa46ded290f7a110"
format_stamp: "Formatted at 2026-09-25 04:39:11 on dist-test-slave-dc95"
server_key: "3a57ed681b84c15968e55528fb981940"
server_key_iv: "4a15f7070013d6200de217e5ebca6dd3"
server_key_version: "encryptionkey@0"
I20260925 04:39:11.951661 5364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311123478858-392-0/minicluster-data/ts-2/wal/instance:
uuid: "76b56ce822c34819aa46ded290f7a110"
format_stamp: "Formatted at 2026-09-25 04:39:11 on dist-test-slave-dc95"
server_key: "3a57ed681b84c15968e55528fb981940"
server_key_iv: "4a15f7070013d6200de217e5ebca6dd3"
server_key_version: "encryptionkey@0"
I20260925 04:39:11.955695 5364 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:39:11.958424 5378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:11.959827 5364 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:39:11.959949 5364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311123478858-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311123478858-392-0/minicluster-data/ts-2/wal
uuid: "76b56ce822c34819aa46ded290f7a110"
format_stamp: "Formatted at 2026-09-25 04:39:11 on dist-test-slave-dc95"
server_key: "3a57ed681b84c15968e55528fb981940"
server_key_iv: "4a15f7070013d6200de217e5ebca6dd3"
server_key_version: "encryptionkey@0"
I20260925 04:39:11.960076 5364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311123478858-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311123478858-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311123478858-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:39:11.965029 5364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:39:11.965687 5364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:39:11.965858 5364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:39:11.966439 5364 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:39:11.967427 5364 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:39:11.967475 5364 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:11.967536 5364 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:39:11.967556 5364 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:11.976320 5364 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44247
I20260925 04:39:11.976346 5491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44247 every 8 connection(s)
I20260925 04:39:11.977242 5364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
I20260925 04:39:11.977945 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 5364
I20260925 04:39:11.978046 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311123478858-392-0/minicluster-data/ts-2/wal/instance
I20260925 04:39:11.978302 392 external_mini_cluster.cc:1428] Setting key 107dc74231aeeb7342cf7f02d1b2336a
I20260925 04:39:11.984119 5081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:11.979121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48451 (local address 127.0.98.62:33223)
0925 04:39:11.979255 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:11.979260 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:11.980786 (+ 1526us) server_negotiation.cc:407] Connection header received
0925 04:39:11.982363 (+ 1577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:11.982366 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:11.982407 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:11.982478 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:11.983628 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:11.983633 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:11.983636 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:39:11.983651 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:11.983677 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:39:11.983688 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:11.983776 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:11.983959 (+ 183us) server_negotiation.cc:299] Negotiation successful
0925 04:39:11.984004 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 04:39:11.985059 5494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:11.979729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33223 (local address 127.0.98.3:48451)
0925 04:39:11.980625 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:11.980683 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:11.982108 (+ 1425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:11.982630 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:11.982637 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:11.982738 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:11.983314 (+ 576us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:39:11.983316 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:11.983360 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:39:11.983434 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:39:11.983800 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:39:11.983809 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:39:11.983814 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 04:39:11.983983 (+ 169us) client_negotiation.cc:240] Negotiation successful
0925 04:39:11.984207 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":176,"threads_started":1}
I20260925 04:39:11.986215 5492 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33223
I20260925 04:39:11.986502 5492 heartbeater.cc:464] Registering TS with master...
I20260925 04:39:11.986941 5492 heartbeater.cc:511] Master 127.0.98.62:33223 requested a full tablet report, sending...
I20260925 04:39:11.988039 5042 ts_manager.cc:194] Registered new tserver with Master: 76b56ce822c34819aa46ded290f7a110 (127.0.98.3:44247)
I20260925 04:39:11.992484 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:39:11.995955 5081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:11.994318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48496 (local address 127.0.98.62:33223)
0925 04:39:11.994425 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:11.994429 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:11.994582 (+ 153us) server_negotiation.cc:407] Connection header received
0925 04:39:11.994700 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:11.994703 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:11.994735 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:11.994792 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:11.995571 (+ 779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:11.995574 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:11.995576 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:39:11.995588 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:11.995644 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:39:11.995655 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:11.995737 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:11.995808 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 04:39:11.995844 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 04:39:11.999073 5042 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48496:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:39:12.002231 5042 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:39:12.015072 5507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:12.012222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44247 (local address 127.0.0.1:35908)
0925 04:39:12.012940 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:12.012952 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:12.013014 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:12.013926 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:12.013929 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:12.013941 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:12.014357 (+ 416us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:39:12.014358 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:12.014367 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:39:12.014397 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:39:12.014752 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:39:12.014755 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:39:12.014759 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 04:39:12.014827 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 04:39:12.014894 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"thread_start_us":101,"threads_started":1}
I20260925 04:39:12.015082 5506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:12.011846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49828 (local address 127.0.98.1:45451)
0925 04:39:12.012239 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:12.012267 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:12.012710 (+ 443us) server_negotiation.cc:407] Connection header received
0925 04:39:12.012773 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:12.012779 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:12.013021 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:12.013149 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:12.014509 (+ 1360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:12.014515 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:12.014527 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:39:12.014538 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:12.014580 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:39:12.014605 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:12.014707 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:12.014834 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 04:39:12.014914 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":104,"threads_started":1}
I20260925 04:39:12.015146 5504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:12.011566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34398 (local address 127.0.98.2:45379)
0925 04:39:12.012073 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:12.012085 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:12.012712 (+ 627us) server_negotiation.cc:407] Connection header received
0925 04:39:12.012772 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:12.012778 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:12.013119 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:12.013253 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:12.014500 (+ 1247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:12.014506 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:12.014520 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:39:12.014534 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:12.014582 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:39:12.014617 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:12.014711 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:12.014926 (+ 215us) server_negotiation.cc:299] Negotiation successful
0925 04:39:12.014997 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":175,"threads_started":1}
I20260925 04:39:12.015344 5505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:12.011670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45451 (local address 127.0.0.1:49828)
0925 04:39:12.012418 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:12.012480 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:12.012605 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:12.013826 (+ 1221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:12.013830 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:12.013844 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:12.014207 (+ 363us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:39:12.014209 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:12.014229 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:39:12.014309 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:39:12.014686 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:39:12.014695 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:39:12.014700 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 04:39:12.014808 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 04:39:12.014917 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"mutex_wait_us":3,"spinlock_wait_cycles":15872,"thread_start_us":109,"threads_started":1}
I20260925 04:39:12.015529 5503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:12.011385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45379 (local address 127.0.0.1:34398)
0925 04:39:12.012442 (+ 1057us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:12.012479 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:12.012605 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:12.013272 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:12.013277 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:12.013356 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:12.013970 (+ 614us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:39:12.013972 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:12.014338 (+ 366us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:39:12.014361 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:39:12.014752 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:39:12.014755 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:39:12.014759 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 04:39:12.014827 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 04:39:12.015302 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":736,"mutex_wait_us":406,"spinlock_wait_cycles":14976,"thread_start_us":106,"threads_started":1}
I20260925 04:39:12.015700 5508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:12.012525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35908 (local address 127.0.98.3:44247)
0925 04:39:12.013031 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:12.013043 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:12.013066 (+ 23us) server_negotiation.cc:407] Connection header received
0925 04:39:12.013114 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:12.013119 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:12.013260 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:12.013370 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:12.014511 (+ 1141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:12.014516 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:12.014524 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:39:12.014535 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:12.014582 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:39:12.014615 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:12.014712 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:12.015417 (+ 705us) server_negotiation.cc:299] Negotiation successful
0925 04:39:12.015500 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":119,"threads_started":1}
I20260925 04:39:12.018333 5295 tablet_service.cc:1467] Processing CreateTablet for tablet 995f5553123c4b619e9b6acbbb815d20 (DEFAULT_TABLE table=test-table [id=a15237d677a74c0a875410e7fc9efee3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:39:12.018471 5426 tablet_service.cc:1467] Processing CreateTablet for tablet 995f5553123c4b619e9b6acbbb815d20 (DEFAULT_TABLE table=test-table [id=a15237d677a74c0a875410e7fc9efee3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:39:12.018481 5164 tablet_service.cc:1467] Processing CreateTablet for tablet 995f5553123c4b619e9b6acbbb815d20 (DEFAULT_TABLE table=test-table [id=a15237d677a74c0a875410e7fc9efee3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:39:12.019430 5295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 995f5553123c4b619e9b6acbbb815d20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:39:12.019484 5426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 995f5553123c4b619e9b6acbbb815d20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:39:12.019538 5164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 995f5553123c4b619e9b6acbbb815d20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:39:12.025987 5509 tablet_bootstrap.cc:492] T 995f5553123c4b619e9b6acbbb815d20 P 987b0a35fc65487ea8cbad5abda7d4e0: Bootstrap starting.
I20260925 04:39:12.027518 5511 tablet_bootstrap.cc:492] T 995f5553123c4b619e9b6acbbb815d20 P 0d273bd980ce43c88852bb7e1db08425: Bootstrap starting.
I20260925 04:39:12.027745 5510 tablet_bootstrap.cc:492] T 995f5553123c4b619e9b6acbbb815d20 P 76b56ce822c34819aa46ded290f7a110: Bootstrap starting.
I20260925 04:39:12.028573 5509 tablet_bootstrap.cc:654] T 995f5553123c4b619e9b6acbbb815d20 P 987b0a35fc65487ea8cbad5abda7d4e0: Neither blocks nor log segments found. Creating new log.
I20260925 04:39:12.029487 5509 log.cc:824] T 995f5553123c4b619e9b6acbbb815d20 P 987b0a35fc65487ea8cbad5abda7d4e0: Log is configured to *not* fsync() on all Append() calls
I20260925 04:39:12.029867 5510 tablet_bootstrap.cc:654] T 995f5553123c4b619e9b6acbbb815d20 P 76b56ce822c34819aa46ded290f7a110: Neither blocks nor log segments found. Creating new log.
I20260925 04:39:12.029866 5511 tablet_bootstrap.cc:654] T 995f5553123c4b619e9b6acbbb815d20 P 0d273bd980ce43c88852bb7e1db08425: Neither blocks nor log segments found. Creating new log.
I20260925 04:39:12.030645 5511 log.cc:824] T 995f5553123c4b619e9b6acbbb815d20 P 0d273bd980ce43c88852bb7e1db08425: Log is configured to *not* fsync() on all Append() calls
I20260925 04:39:12.030711 5510 log.cc:824] T 995f5553123c4b619e9b6acbbb815d20 P 76b56ce822c34819aa46ded290f7a110: Log is configured to *not* fsync() on all Append() calls
I20260925 04:39:12.031888 5509 tablet_bootstrap.cc:492] T 995f5553123c4b619e9b6acbbb815d20 P 987b0a35fc65487ea8cbad5abda7d4e0: No bootstrap required, opened a new log
I20260925 04:39:12.032085 5509 ts_tablet_manager.cc:1397] T 995f5553123c4b619e9b6acbbb815d20 P 987b0a35fc65487ea8cbad5abda7d4e0: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 04:39:12.032692 5510 tablet_bootstrap.cc:492] T 995f5553123c4b619e9b6acbbb815d20 P 76b56ce822c34819aa46ded290f7a110: No bootstrap required, opened a new log
I20260925 04:39:12.032692 5511 tablet_bootstrap.cc:492] T 995f5553123c4b619e9b6acbbb815d20 P 0d273bd980ce43c88852bb7e1db08425: No bootstrap required, opened a new log
I20260925 04:39:12.032886 5511 ts_tablet_manager.cc:1397] T 995f5553123c4b619e9b6acbbb815d20 P 0d273bd980ce43c88852bb7e1db08425: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260925 04:39:12.032895 5510 ts_tablet_manager.cc:1397] T 995f5553123c4b619e9b6acbbb815d20 P 76b56ce822c34819aa46ded290f7a110: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 04:39:12.036104 5509 raft_consensus.cc:348] T 995f5553123c4b619e9b6acbbb815d20 P 987b0a35fc65487ea8cbad5abda7d4e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987b0a35fc65487ea8cbad5abda7d4e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45379 } } peers { permanent_uuid: "0d273bd980ce43c88852bb7e1db08425" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45451 } } peers { permanent_uuid: "76b56ce822c34819aa46ded290f7a110" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44247 } }
I20260925 04:39:12.036358 5509 raft_consensus.cc:374] T 995f5553123c4b619e9b6acbbb815d20 P 987b0a35fc65487ea8cbad5abda7d4e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:39:12.036412 5509 raft_consensus.cc:729] T 995f5553123c4b619e9b6acbbb815d20 P 987b0a35fc65487ea8cbad5abda7d4e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 987b0a35fc65487ea8cbad5abda7d4e0, State: Initialized, Role: FOLLOWER
I20260925 04:39:12.036859 5509 consensus_queue.cc:260] T 995f5553123c4b619e9b6acbbb815d20 P 987b0a35fc65487ea8cbad5abda7d4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987b0a35fc65487ea8cbad5abda7d4e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45379 } } peers { permanent_uuid: "0d273bd980ce43c88852bb7e1db08425" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45451 } } peers { permanent_uuid: "76b56ce822c34819aa46ded290f7a110" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44247 } }
I20260925 04:39:12.037560 5361 heartbeater.cc:503] Master 127.0.98.62:33223 was elected leader, sending a full tablet report...
I20260925 04:39:12.037429 5511 raft_consensus.cc:348] T 995f5553123c4b619e9b6acbbb815d20 P 0d273bd980ce43c88852bb7e1db08425 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987b0a35fc65487ea8cbad5abda7d4e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45379 } } peers { permanent_uuid: "0d273bd980ce43c88852bb7e1db08425" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45451 } } peers { permanent_uuid: "76b56ce822c34819aa46ded290f7a110" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44247 } }
I20260925 04:39:12.037429 5510 raft_consensus.cc:348] T 995f5553123c4b619e9b6acbbb815d20 P 76b56ce822c34819aa46ded290f7a110 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987b0a35fc65487ea8cbad5abda7d4e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45379 } } peers { permanent_uuid: "0d273bd980ce43c88852bb7e1db08425" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45451 } } peers { permanent_uuid: "76b56ce822c34819aa46ded290f7a110" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44247 } }
I20260925 04:39:12.037707 5511 raft_consensus.cc:374] T 995f5553123c4b619e9b6acbbb815d20 P 0d273bd980ce43c88852bb7e1db08425 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:39:12.037707 5510 raft_consensus.cc:374] T 995f5553123c4b619e9b6acbbb815d20 P 76b56ce822c34819aa46ded290f7a110 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:39:12.037755 5510 raft_consensus.cc:729] T 995f5553123c4b619e9b6acbbb815d20 P 76b56ce822c34819aa46ded290f7a110 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76b56ce822c34819aa46ded290f7a110, State: Initialized, Role: FOLLOWER
I20260925 04:39:12.037755 5511 raft_consensus.cc:729] T 995f5553123c4b619e9b6acbbb815d20 P 0d273bd980ce43c88852bb7e1db08425 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d273bd980ce43c88852bb7e1db08425, State: Initialized, Role: FOLLOWER
I20260925 04:39:12.037974 5509 ts_tablet_manager.cc:1428] T 995f5553123c4b619e9b6acbbb815d20 P 987b0a35fc65487ea8cbad5abda7d4e0: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 04:39:12.038280 5511 consensus_queue.cc:260] T 995f5553123c4b619e9b6acbbb815d20 P 0d273bd980ce43c88852bb7e1db08425 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987b0a35fc65487ea8cbad5abda7d4e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45379 } } peers { permanent_uuid: "0d273bd980ce43c88852bb7e1db08425" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45451 } } peers { permanent_uuid: "76b56ce822c34819aa46ded290f7a110" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44247 } }
I20260925 04:39:12.038280 5510 consensus_queue.cc:260] T 995f5553123c4b619e9b6acbbb815d20 P 76b56ce822c34819aa46ded290f7a110 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987b0a35fc65487ea8cbad5abda7d4e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45379 } } peers { permanent_uuid: "0d273bd980ce43c88852bb7e1db08425" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45451 } } peers { permanent_uuid: "76b56ce822c34819aa46ded290f7a110" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44247 } }
I20260925 04:39:12.039219 5230 heartbeater.cc:503] Master 127.0.98.62:33223 was elected leader, sending a full tablet report...
I20260925 04:39:12.039340 5511 ts_tablet_manager.cc:1428] T 995f5553123c4b619e9b6acbbb815d20 P 0d273bd980ce43c88852bb7e1db08425: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 04:39:12.039340 5510 ts_tablet_manager.cc:1428] T 995f5553123c4b619e9b6acbbb815d20 P 76b56ce822c34819aa46ded290f7a110: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260925 04:39:12.039857 5492 heartbeater.cc:503] Master 127.0.98.62:33223 was elected leader, sending a full tablet report...
W20260925 04:39:12.045674 5362 tablet.cc:2369] T 995f5553123c4b619e9b6acbbb815d20 P 987b0a35fc65487ea8cbad5abda7d4e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 04:39:12.112506 5231 tablet.cc:2369] T 995f5553123c4b619e9b6acbbb815d20 P 0d273bd980ce43c88852bb7e1db08425: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:39:12.168004 5517 raft_consensus.cc:482] T 995f5553123c4b619e9b6acbbb815d20 P 76b56ce822c34819aa46ded290f7a110 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:39:12.168227 5517 raft_consensus.cc:504] T 995f5553123c4b619e9b6acbbb815d20 P 76b56ce822c34819aa46ded290f7a110 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987b0a35fc65487ea8cbad5abda7d4e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45379 } } peers { permanent_uuid: "0d273bd980ce43c88852bb7e1db08425" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45451 } } peers { permanent_uuid: "76b56ce822c34819aa46ded290f7a110" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44247 } }
I20260925 04:39:12.169409 5517 leader_election.cc:288] T 995f5553123c4b619e9b6acbbb815d20 P 76b56ce822c34819aa46ded290f7a110 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 987b0a35fc65487ea8cbad5abda7d4e0 (127.0.98.2:45379), 0d273bd980ce43c88852bb7e1db08425 (127.0.98.1:45451)
I20260925 04:39:12.171418 5504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:12.169755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39645 (local address 127.0.98.2:45379)
0925 04:39:12.169890 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:12.169894 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:12.170062 (+ 168us) server_negotiation.cc:407] Connection header received
0925 04:39:12.170118 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:12.170122 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:12.170165 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:12.170246 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:12.171008 (+ 762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:12.171011 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:12.171014 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:39:12.171026 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:12.171049 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:39:12.171059 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:12.171143 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:12.171234 (+ 91us) server_negotiation.cc:299] Negotiation successful
0925 04:39:12.171278 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 04:39:12.171418 5494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:12.169664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45379 (local address 127.0.98.3:39645)
0925 04:39:12.169867 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:12.169884 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:12.169991 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:12.170325 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:12.170328 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:12.170342 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:12.170844 (+ 502us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:39:12.170846 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:12.170856 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:39:12.170881 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:39:12.171134 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:39:12.171137 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:39:12.171140 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 04:39:12.171219 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 04:39:12.171263 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120}
I20260925 04:39:12.171511 5506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:12.169898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36929 (local address 127.0.98.1:45451)
0925 04:39:12.170018 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:12.170022 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:12.170186 (+ 164us) server_negotiation.cc:407] Connection header received
0925 04:39:12.170285 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:12.170289 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:12.170329 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:12.170413 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:12.171106 (+ 693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:12.171110 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:12.171113 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:39:12.171125 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:12.171146 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:39:12.171156 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:12.171233 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:12.171361 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 04:39:12.171401 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 04:39:12.171746 5518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:12.169710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45451 (local address 127.0.98.3:36929)
0925 04:39:12.170101 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:12.170114 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:12.170204 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:12.170439 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:12.170442 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:12.170454 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:12.170956 (+ 502us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:39:12.170958 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:12.170969 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:39:12.170996 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:39:12.171229 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:39:12.171232 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:39:12.171235 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 04:39:12.171355 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 04:39:12.171386 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":129,"threads_started":1}
I20260925 04:39:12.172144 5315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "995f5553123c4b619e9b6acbbb815d20" candidate_uuid: "76b56ce822c34819aa46ded290f7a110" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "987b0a35fc65487ea8cbad5abda7d4e0" is_pre_election: true
I20260925 04:39:12.172248 5184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "995f5553123c4b619e9b6acbbb815d20" candidate_uuid: "76b56ce822c34819aa46ded290f7a110" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d273bd980ce43c88852bb7e1db08425" is_pre_election: true
I20260925 04:39:12.172472 5315 raft_consensus.cc:2454] T 995f5553123c4b619e9b6acbbb815d20 P 987b0a35fc65487ea8cbad5abda7d4e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76b56ce822c34819aa46ded290f7a110 in term 0.
I20260925 04:39:12.172533 5184 raft_consensus.cc:2454] T 995f5553123c4b619e9b6acbbb815d20 P 0d273bd980ce43c88852bb7e1db08425 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76b56ce822c34819aa46ded290f7a110 in term 0.
I20260925 04:39:12.172960 5381 leader_election.cc:302] T 995f5553123c4b619e9b6acbbb815d20 P 76b56ce822c34819aa46ded290f7a110 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d273bd980ce43c88852bb7e1db08425, 76b56ce822c34819aa46ded290f7a110; no voters:
I20260925 04:39:12.173228 5517 raft_consensus.cc:2781] T 995f5553123c4b619e9b6acbbb815d20 P 76b56ce822c34819aa46ded290f7a110 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:39:12.173312 5517 raft_consensus.cc:482] T 995f5553123c4b619e9b6acbbb815d20 P 76b56ce822c34819aa46ded290f7a110 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:39:12.173344 5517 raft_consensus.cc:3037] T 995f5553123c4b619e9b6acbbb815d20 P 76b56ce822c34819aa46ded290f7a110 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:39:12.174347 5517 raft_consensus.cc:504] T 995f5553123c4b619e9b6acbbb815d20 P 76b56ce822c34819aa46ded290f7a110 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987b0a35fc65487ea8cbad5abda7d4e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45379 } } peers { permanent_uuid: "0d273bd980ce43c88852bb7e1db08425" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45451 } } peers { permanent_uuid: "76b56ce822c34819aa46ded290f7a110" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44247 } }
I20260925 04:39:12.174711 5517 leader_election.cc:288] T 995f5553123c4b619e9b6acbbb815d20 P 76b56ce822c34819aa46ded290f7a110 [CANDIDATE]: Term 1 election: Requested vote from peers 987b0a35fc65487ea8cbad5abda7d4e0 (127.0.98.2:45379), 0d273bd980ce43c88852bb7e1db08425 (127.0.98.1:45451)
I20260925 04:39:12.175086 5184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "995f5553123c4b619e9b6acbbb815d20" candidate_uuid: "76b56ce822c34819aa46ded290f7a110" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d273bd980ce43c88852bb7e1db08425"
I20260925 04:39:12.175102 5315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "995f5553123c4b619e9b6acbbb815d20" candidate_uuid: "76b56ce822c34819aa46ded290f7a110" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "987b0a35fc65487ea8cbad5abda7d4e0"
I20260925 04:39:12.175202 5184 raft_consensus.cc:3037] T 995f5553123c4b619e9b6acbbb815d20 P 0d273bd980ce43c88852bb7e1db08425 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:39:12.175220 5315 raft_consensus.cc:3037] T 995f5553123c4b619e9b6acbbb815d20 P 987b0a35fc65487ea8cbad5abda7d4e0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:39:12.176319 5315 raft_consensus.cc:2454] T 995f5553123c4b619e9b6acbbb815d20 P 987b0a35fc65487ea8cbad5abda7d4e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76b56ce822c34819aa46ded290f7a110 in term 1.
I20260925 04:39:12.176340 5184 raft_consensus.cc:2454] T 995f5553123c4b619e9b6acbbb815d20 P 0d273bd980ce43c88852bb7e1db08425 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76b56ce822c34819aa46ded290f7a110 in term 1.
I20260925 04:39:12.176614 5381 leader_election.cc:302] T 995f5553123c4b619e9b6acbbb815d20 P 76b56ce822c34819aa46ded290f7a110 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d273bd980ce43c88852bb7e1db08425, 76b56ce822c34819aa46ded290f7a110; no voters:
I20260925 04:39:12.176772 5517 raft_consensus.cc:2781] T 995f5553123c4b619e9b6acbbb815d20 P 76b56ce822c34819aa46ded290f7a110 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:39:12.177027 5517 raft_consensus.cc:686] T 995f5553123c4b619e9b6acbbb815d20 P 76b56ce822c34819aa46ded290f7a110 [term 1 LEADER]: Becoming Leader. State: Replica: 76b56ce822c34819aa46ded290f7a110, State: Running, Role: LEADER
I20260925 04:39:12.177389 5517 consensus_queue.cc:237] T 995f5553123c4b619e9b6acbbb815d20 P 76b56ce822c34819aa46ded290f7a110 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987b0a35fc65487ea8cbad5abda7d4e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45379 } } peers { permanent_uuid: "0d273bd980ce43c88852bb7e1db08425" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45451 } } peers { permanent_uuid: "76b56ce822c34819aa46ded290f7a110" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44247 } }
I20260925 04:39:12.180588 5042 catalog_manager.cc:5495] T 995f5553123c4b619e9b6acbbb815d20 P 76b56ce822c34819aa46ded290f7a110 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76b56ce822c34819aa46ded290f7a110 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "76b56ce822c34819aa46ded290f7a110" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987b0a35fc65487ea8cbad5abda7d4e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d273bd980ce43c88852bb7e1db08425" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76b56ce822c34819aa46ded290f7a110" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44247 } health_report { overall_health: HEALTHY } } }
I20260925 04:39:12.206679 5508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:12.205204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35924 (local address 127.0.98.3:44247)
0925 04:39:12.205318 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:12.205322 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:12.205355 (+ 33us) server_negotiation.cc:407] Connection header received
0925 04:39:12.205433 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:12.205435 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:12.205495 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:12.205557 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:12.206305 (+ 748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:12.206309 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:12.206311 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:39:12.206323 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:12.206344 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:39:12.206352 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:12.206434 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:12.206531 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 04:39:12.206567 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 04:39:12.213122 5184 raft_consensus.cc:1261] T 995f5553123c4b619e9b6acbbb815d20 P 0d273bd980ce43c88852bb7e1db08425 [term 1 FOLLOWER]: Refusing update from remote peer 76b56ce822c34819aa46ded290f7a110: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:39:12.213122 5315 raft_consensus.cc:1261] T 995f5553123c4b619e9b6acbbb815d20 P 987b0a35fc65487ea8cbad5abda7d4e0 [term 1 FOLLOWER]: Refusing update from remote peer 76b56ce822c34819aa46ded290f7a110: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:39:12.213930 5519 consensus_queue.cc:1035] T 995f5553123c4b619e9b6acbbb815d20 P 76b56ce822c34819aa46ded290f7a110 [LEADER]: Connected to new peer: Peer: permanent_uuid: "987b0a35fc65487ea8cbad5abda7d4e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:39:12.214165 5517 consensus_queue.cc:1035] T 995f5553123c4b619e9b6acbbb815d20 P 76b56ce822c34819aa46ded290f7a110 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d273bd980ce43c88852bb7e1db08425" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:39:12.222544 5523 mvcc.cc:205] Tried to move back new op lower bound from 7333114479456677888 to 7333114479321296896. Current Snapshot: MvccSnapshot[applied={T|T < 7333114479456677888}]
I20260925 04:39:12.224860 5524 mvcc.cc:205] Tried to move back new op lower bound from 7333114479456677888 to 7333114479321296896. Current Snapshot: MvccSnapshot[applied={T|T < 7333114479456677888}]
I20260925 04:39:12.231783 5041 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48496:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:39:12.231973 5041 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48496:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:39:12.234635 5041 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 53abdf3881c649e0b30cd244403f464f: Sending DeleteTablet for 3 replicas of tablet 995f5553123c4b619e9b6acbbb815d20
I20260925 04:39:12.235628 5426 tablet_service.cc:1514] Processing DeleteTablet for tablet 995f5553123c4b619e9b6acbbb815d20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:39:12 UTC) from {username='slave'} at 127.0.0.1:35908
I20260925 04:39:12.235636 5295 tablet_service.cc:1514] Processing DeleteTablet for tablet 995f5553123c4b619e9b6acbbb815d20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:39:12 UTC) from {username='slave'} at 127.0.0.1:34398
I20260925 04:39:12.235641 5164 tablet_service.cc:1514] Processing DeleteTablet for tablet 995f5553123c4b619e9b6acbbb815d20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:39:12 UTC) from {username='slave'} at 127.0.0.1:49828
I20260925 04:39:12.236250 5532 tablet_replica.cc:330] T 995f5553123c4b619e9b6acbbb815d20 P 987b0a35fc65487ea8cbad5abda7d4e0: stopping tablet replica
I20260925 04:39:12.236632 5532 raft_consensus.cc:2229] T 995f5553123c4b619e9b6acbbb815d20 P 987b0a35fc65487ea8cbad5abda7d4e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:39:12.236642 5533 tablet_replica.cc:330] T 995f5553123c4b619e9b6acbbb815d20 P 76b56ce822c34819aa46ded290f7a110: stopping tablet replica
I20260925 04:39:12.236784 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 5102
I20260925 04:39:12.236940 5533 raft_consensus.cc:2229] T 995f5553123c4b619e9b6acbbb815d20 P 76b56ce822c34819aa46ded290f7a110 [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:39:12.236984 5532 raft_consensus.cc:2258] T 995f5553123c4b619e9b6acbbb815d20 P 987b0a35fc65487ea8cbad5abda7d4e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:39:12.237556 5533 raft_consensus.cc:2258] T 995f5553123c4b619e9b6acbbb815d20 P 76b56ce822c34819aa46ded290f7a110 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:39:12.238996 5532 ts_tablet_manager.cc:1905] T 995f5553123c4b619e9b6acbbb815d20 P 987b0a35fc65487ea8cbad5abda7d4e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:39:12.241109 5533 ts_tablet_manager.cc:1905] T 995f5553123c4b619e9b6acbbb815d20 P 76b56ce822c34819aa46ded290f7a110: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:39:12.242524 5532 ts_tablet_manager.cc:1918] T 995f5553123c4b619e9b6acbbb815d20 P 987b0a35fc65487ea8cbad5abda7d4e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:39:12.242671 5532 log.cc:1196] T 995f5553123c4b619e9b6acbbb815d20 P 987b0a35fc65487ea8cbad5abda7d4e0: Deleting WAL directory at /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311123478858-392-0/minicluster-data/ts-1/wal/wals/995f5553123c4b619e9b6acbbb815d20
I20260925 04:39:12.243034 5532 ts_tablet_manager.cc:1939] T 995f5553123c4b619e9b6acbbb815d20 P 987b0a35fc65487ea8cbad5abda7d4e0: Deleting consensus metadata
I20260925 04:39:12.244709 5533 ts_tablet_manager.cc:1918] T 995f5553123c4b619e9b6acbbb815d20 P 76b56ce822c34819aa46ded290f7a110: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:39:12.244922 5533 log.cc:1196] T 995f5553123c4b619e9b6acbbb815d20 P 76b56ce822c34819aa46ded290f7a110: Deleting WAL directory at /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311123478858-392-0/minicluster-data/ts-2/wal/wals/995f5553123c4b619e9b6acbbb815d20
I20260925 04:39:12.244977 5027 catalog_manager.cc:4841] TS 987b0a35fc65487ea8cbad5abda7d4e0 (127.0.98.2:45379): tablet 995f5553123c4b619e9b6acbbb815d20 (table test-table [id=a15237d677a74c0a875410e7fc9efee3]) successfully deleted
W20260925 04:39:12.245133 5028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45451 (error 108)
I20260925 04:39:12.245388 5533 ts_tablet_manager.cc:1939] T 995f5553123c4b619e9b6acbbb815d20 P 76b56ce822c34819aa46ded290f7a110: Deleting consensus metadata
I20260925 04:39:12.246075 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 5233
I20260925 04:39:12.246907 5503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:12.246370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45451 (local address 127.0.0.1:49834)
0925 04:39:12.246738 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:12.246810 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":283}
W20260925 04:39:12.247534 5028 catalog_manager.cc:4584] TS 0d273bd980ce43c88852bb7e1db08425 (127.0.98.1:45451): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 995f5553123c4b619e9b6acbbb815d20: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45451: connect: Connection refused (error 111)
I20260925 04:39:12.247786 5028 catalog_manager.cc:4841] TS 76b56ce822c34819aa46ded290f7a110 (127.0.98.3:44247): tablet 995f5553123c4b619e9b6acbbb815d20 (table test-table [id=a15237d677a74c0a875410e7fc9efee3]) successfully deleted
I20260925 04:39:12.251806 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 5364
I20260925 04:39:12.257835 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 5011
2026-09-25T04:39:12Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1099 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260925 04:39:12.278230 392 mini_oidc.cc:152] Starting JWKS server
I20260925 04:39:12.279842 392 webserver.cc:460] Webserver started at https://127.0.0.1:36401/ using document root <none> and password file <none>
I20260925 04:39:12.280005 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 04:39:12.280284 392 webserver.cc:460] Webserver started at http://127.0.0.1:46853/ using document root <none> and password file <none>
2026-09-25T04:39:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:39:12Z Disabled control of system clock
I20260925 04:39:12.290009 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790311123478858-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790311123478858-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790311123478858-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35195
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:32895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35195
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36401/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790311123478858-392-0/testchainca.cert with env {}
W20260925 04:39:12.406325 5541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:39:12.406651 5541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:39:12.406695 5541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:39:12.411005 5541 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:39:12.411106 5541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:39:12.411121 5541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:39:12.411137 5541 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:39:12.411152 5541 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 04:39:12.411168 5541 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 04:39:12.411180 5541 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36401/jwks.json
I20260925 04:39:12.417460 5541 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790311123478858-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790311123478858-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35195
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36401/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790311123478858-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790311123478858-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:39:12.418602 5541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:39:12.419655 5541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:39:12.428607 5549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:12.428611 5547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:12.428754 5541 server_base.cc:1034] running on GCE node
W20260925 04:39:12.428607 5546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:12.429544 5541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:39:12.430814 5541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:39:12.432077 5541 hybrid_clock.cc:648] HybridClock initialized: now 1790311152432054 us; error 42 us; skew 500 ppm
I20260925 04:39:12.434283 5541 webserver.cc:460] Webserver started at http://127.0.98.62:44759/ using document root <none> and password file <none>
I20260925 04:39:12.434906 5541 fs_manager.cc:359] Metadata directory not provided
I20260925 04:39:12.434988 5541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:39:12.435204 5541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:39:12.437481 5541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790311123478858-392-0/minicluster-data/master-0/data/instance:
uuid: "33c2435b6d0747b389d2737d3e1aeee1"
format_stamp: "Formatted at 2026-09-25 04:39:12 on dist-test-slave-dc95"
server_key: "7bdef52e1b1d93664e5710c7b7c629e5"
server_key_iv: "a8be5bdab38b988472317ef6ae66376a"
server_key_version: "encryptionkey@0"
I20260925 04:39:12.438066 5541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790311123478858-392-0/minicluster-data/master-0/wal/instance:
uuid: "33c2435b6d0747b389d2737d3e1aeee1"
format_stamp: "Formatted at 2026-09-25 04:39:12 on dist-test-slave-dc95"
server_key: "7bdef52e1b1d93664e5710c7b7c629e5"
server_key_iv: "a8be5bdab38b988472317ef6ae66376a"
server_key_version: "encryptionkey@0"
I20260925 04:39:12.442142 5541 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260925 04:39:12.445222 5555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:12.446786 5541 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 04:39:12.446944 5541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790311123478858-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790311123478858-392-0/minicluster-data/master-0/wal
uuid: "33c2435b6d0747b389d2737d3e1aeee1"
format_stamp: "Formatted at 2026-09-25 04:39:12 on dist-test-slave-dc95"
server_key: "7bdef52e1b1d93664e5710c7b7c629e5"
server_key_iv: "a8be5bdab38b988472317ef6ae66376a"
server_key_version: "encryptionkey@0"
I20260925 04:39:12.447075 5541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790311123478858-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790311123478858-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790311123478858-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:39:12.464931 5541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:39:12.465716 5541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:39:12.465922 5541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:39:12.472927 5541 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35195
I20260925 04:39:12.472936 5607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35195 every 8 connection(s)
I20260925 04:39:12.473940 5541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
I20260925 04:39:12.476642 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 5541
I20260925 04:39:12.476744 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790311123478858-392-0/minicluster-data/master-0/wal/instance
I20260925 04:39:12.477026 392 external_mini_cluster.cc:1428] Setting key 51f4df043137b94c647d3aed9dec03cf
I20260925 04:39:12.477545 5608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:39:12.485435 5608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33c2435b6d0747b389d2737d3e1aeee1: Bootstrap starting.
I20260925 04:39:12.488175 5611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:12.478544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60110 (local address 127.0.98.62:35195)
0925 04:39:12.479268 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:12.479284 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:12.479325 (+ 41us) server_negotiation.cc:407] Connection header received
0925 04:39:12.480120 (+ 795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:12.480147 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:12.480721 (+ 574us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:12.481160 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:12.481659 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:12.483075 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:12.484076 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:12.484802 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:12.484962 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:12.485842 (+ 880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:12.485848 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:12.485867 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:39:12.486391 (+ 524us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:12.486465 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:39:12.486495 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:12.486597 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:12.486777 (+ 180us) server_negotiation.cc:299] Negotiation successful
0925 04:39:12.487103 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":155,"threads_started":1}
I20260925 04:39:12.488458 5608 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33c2435b6d0747b389d2737d3e1aeee1: Neither blocks nor log segments found. Creating new log.
I20260925 04:39:12.489401 5608 log.cc:824] T 00000000000000000000000000000000 P 33c2435b6d0747b389d2737d3e1aeee1: Log is configured to *not* fsync() on all Append() calls
I20260925 04:39:12.491708 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:39:12.491916 5608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33c2435b6d0747b389d2737d3e1aeee1: No bootstrap required, opened a new log
I20260925 04:39:12.495731 5608 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33c2435b6d0747b389d2737d3e1aeee1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c2435b6d0747b389d2737d3e1aeee1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35195 } }
I20260925 04:39:12.496074 5608 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33c2435b6d0747b389d2737d3e1aeee1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:39:12.496129 5608 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33c2435b6d0747b389d2737d3e1aeee1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33c2435b6d0747b389d2737d3e1aeee1, State: Initialized, Role: FOLLOWER
I20260925 04:39:12.496604 5608 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33c2435b6d0747b389d2737d3e1aeee1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c2435b6d0747b389d2737d3e1aeee1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35195 } }
I20260925 04:39:12.496743 5608 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33c2435b6d0747b389d2737d3e1aeee1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:39:12.496796 5608 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33c2435b6d0747b389d2737d3e1aeee1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:39:12.496891 5608 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33c2435b6d0747b389d2737d3e1aeee1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:39:12.497984 5608 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33c2435b6d0747b389d2737d3e1aeee1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c2435b6d0747b389d2737d3e1aeee1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35195 } }
I20260925 04:39:12.498317 5608 leader_election.cc:302] T 00000000000000000000000000000000 P 33c2435b6d0747b389d2737d3e1aeee1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33c2435b6d0747b389d2737d3e1aeee1; no voters:
I20260925 04:39:12.498600 5608 leader_election.cc:288] T 00000000000000000000000000000000 P 33c2435b6d0747b389d2737d3e1aeee1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:39:12.498723 5613 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33c2435b6d0747b389d2737d3e1aeee1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:39:12.498937 5613 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33c2435b6d0747b389d2737d3e1aeee1 [term 1 LEADER]: Becoming Leader. State: Replica: 33c2435b6d0747b389d2737d3e1aeee1, State: Running, Role: LEADER
I20260925 04:39:12.499254 5613 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33c2435b6d0747b389d2737d3e1aeee1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c2435b6d0747b389d2737d3e1aeee1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35195 } }
I20260925 04:39:12.499931 5608 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33c2435b6d0747b389d2737d3e1aeee1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:39:12.501017 5614 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33c2435b6d0747b389d2737d3e1aeee1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33c2435b6d0747b389d2737d3e1aeee1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c2435b6d0747b389d2737d3e1aeee1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35195 } } }
I20260925 04:39:12.501154 5614 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33c2435b6d0747b389d2737d3e1aeee1 [sys.catalog]: This master's current role is: LEADER
I20260925 04:39:12.501413 5615 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33c2435b6d0747b389d2737d3e1aeee1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33c2435b6d0747b389d2737d3e1aeee1. Latest consensus state: current_term: 1 leader_uuid: "33c2435b6d0747b389d2737d3e1aeee1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c2435b6d0747b389d2737d3e1aeee1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35195 } } }
I20260925 04:39:12.501533 5615 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33c2435b6d0747b389d2737d3e1aeee1 [sys.catalog]: This master's current role is: LEADER
I20260925 04:39:12.501619 5622 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:39:12.504531 5622 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:39:12.511030 5622 catalog_manager.cc:1347] Generated new cluster ID: 94aa1ac570a441aca330f109592a4f3e
I20260925 04:39:12.511114 5622 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:39:12.524739 5622 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:39:12.525754 5622 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:39:12.542644 5622 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33c2435b6d0747b389d2737d3e1aeee1: Generated new TSK 0
I20260925 04:39:12.543426 5622 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 04:39:12.569276 5638 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260925 04:39:12.569382 5638 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 04:39:12.570058 5611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:12.552404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60118 (local address 127.0.98.62:35195)
0925 04:39:12.552541 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:12.552546 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:12.552697 (+ 151us) server_negotiation.cc:407] Connection header received
0925 04:39:12.552894 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:12.552898 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:12.552961 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:12.553076 (+ 115us) server_negotiation.cc:226] Negotiated authn=JWT
0925 04:39:12.553281 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:12.554102 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:12.555015 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:12.555346 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:12.555445 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:12.555798 (+ 353us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 04:39:12.569603 (+ 13805us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36401/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 04:39:12.569886 (+ 283us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60118: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36401/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":44}
W20260925 04:39:12.570127 5611 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60118: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36401/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260925 04:39:12.571225 5637 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:39:12.552283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35195 (local address 127.0.0.1:60118)
0925 04:39:12.552622 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:12.552635 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:12.552748 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:12.553082 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:12.553085 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:12.553091 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0925 04:39:12.553191 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:12.553196 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:12.554227 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:12.554229 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:12.554878 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:12.554887 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:12.555443 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:12.555463 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:12.555525 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:12.555575 (+ 50us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 04:39:12.569857 (+ 14282us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36401/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 04:39:12.570070 (+ 213us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:35195: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36401/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":131,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:35195: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36401/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260925 04:39:12.571902 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 5541
2026-09-25T04:39:12Z chronyd exiting
I20260925 04:39:12.608631 392 test_util.cc:175] -----------------------------------------------
I20260925 04:39:12.608723 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790311123478858-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (332 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260925 04:39:12.609787 392 mini_oidc.cc:152] Starting JWKS server
I20260925 04:39:12.610631 392 webserver.cc:460] Webserver started at https://127.0.0.1:45645/ using document root <none> and password file <none>
I20260925 04:39:12.610719 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 04:39:12.610971 392 webserver.cc:460] Webserver started at http://127.0.0.1:36447/ using document root <none> and password file <none>
2026-09-25T04:39:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:39:12Z Disabled control of system clock
I20260925 04:39:12.619454 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790311123478858-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790311123478858-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790311123478858-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35043
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35043
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45645/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790311123478858-392-0/testchainca.cert with env {}
W20260925 04:39:12.730487 5644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:39:12.730813 5644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:39:12.730872 5644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:39:12.735210 5644 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:39:12.735291 5644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:39:12.735313 5644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:39:12.735335 5644 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:39:12.735354 5644 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 04:39:12.735385 5644 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 04:39:12.735411 5644 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45645/jwks.json
I20260925 04:39:12.741173 5644 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790311123478858-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790311123478858-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35043
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45645/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790311123478858-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790311123478858-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:39:12.742307 5644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:39:12.743368 5644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:39:12.752607 5652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:12.752599 5650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:12.752696 5644 server_base.cc:1034] running on GCE node
W20260925 04:39:12.752599 5649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:12.753481 5644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:39:12.754781 5644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:39:12.755973 5644 hybrid_clock.cc:648] HybridClock initialized: now 1790311152755952 us; error 39 us; skew 500 ppm
I20260925 04:39:12.758215 5644 webserver.cc:460] Webserver started at http://127.0.98.62:41601/ using document root <none> and password file <none>
I20260925 04:39:12.758919 5644 fs_manager.cc:359] Metadata directory not provided
I20260925 04:39:12.758999 5644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:39:12.759204 5644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:39:12.761444 5644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790311123478858-392-0/minicluster-data/master-0/data/instance:
uuid: "e689031926364272be42e55905f4223d"
format_stamp: "Formatted at 2026-09-25 04:39:12 on dist-test-slave-dc95"
server_key: "fa49b512551f957a54609e9124229e3c"
server_key_iv: "9fba797e1c8c969f288753df296396fb"
server_key_version: "encryptionkey@0"
I20260925 04:39:12.762091 5644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790311123478858-392-0/minicluster-data/master-0/wal/instance:
uuid: "e689031926364272be42e55905f4223d"
format_stamp: "Formatted at 2026-09-25 04:39:12 on dist-test-slave-dc95"
server_key: "fa49b512551f957a54609e9124229e3c"
server_key_iv: "9fba797e1c8c969f288753df296396fb"
server_key_version: "encryptionkey@0"
I20260925 04:39:12.766254 5644 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260925 04:39:12.769116 5658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:12.770637 5644 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 04:39:12.770787 5644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790311123478858-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790311123478858-392-0/minicluster-data/master-0/wal
uuid: "e689031926364272be42e55905f4223d"
format_stamp: "Formatted at 2026-09-25 04:39:12 on dist-test-slave-dc95"
server_key: "fa49b512551f957a54609e9124229e3c"
server_key_iv: "9fba797e1c8c969f288753df296396fb"
server_key_version: "encryptionkey@0"
I20260925 04:39:12.770912 5644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790311123478858-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790311123478858-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790311123478858-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:39:12.781200 5644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:39:12.781941 5644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:39:12.782126 5644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:39:12.789284 5644 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35043
I20260925 04:39:12.789284 5710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35043 every 8 connection(s)
I20260925 04:39:12.790328 5644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
I20260925 04:39:12.793586 5711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:39:12.795301 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 5644
I20260925 04:39:12.795406 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790311123478858-392-0/minicluster-data/master-0/wal/instance
I20260925 04:39:12.795787 392 external_mini_cluster.cc:1428] Setting key d0639f387f35bf507e4ab4bb0e08b416
I20260925 04:39:12.800316 5711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e689031926364272be42e55905f4223d: Bootstrap starting.
I20260925 04:39:12.803004 5711 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e689031926364272be42e55905f4223d: Neither blocks nor log segments found. Creating new log.
I20260925 04:39:12.803876 5711 log.cc:824] T 00000000000000000000000000000000 P e689031926364272be42e55905f4223d: Log is configured to *not* fsync() on all Append() calls
I20260925 04:39:12.805264 5714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:12.797206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52894 (local address 127.0.98.62:35043)
0925 04:39:12.797852 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:12.797868 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:12.797904 (+ 36us) server_negotiation.cc:407] Connection header received
0925 04:39:12.798588 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:12.798607 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:12.799032 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:12.799319 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:12.799713 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:12.800725 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:12.801858 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:12.802344 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:12.802475 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:12.803410 (+ 935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:12.803415 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:12.803429 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:39:12.803871 (+ 442us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:12.803971 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:39:12.804001 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:12.804095 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:12.804243 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 04:39:12.804522 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":127,"threads_started":1}
I20260925 04:39:12.805889 5711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e689031926364272be42e55905f4223d: No bootstrap required, opened a new log
I20260925 04:39:12.808146 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:39:12.810006 5711 raft_consensus.cc:348] T 00000000000000000000000000000000 P e689031926364272be42e55905f4223d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e689031926364272be42e55905f4223d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35043 } }
I20260925 04:39:12.810343 5711 raft_consensus.cc:374] T 00000000000000000000000000000000 P e689031926364272be42e55905f4223d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:39:12.810405 5711 raft_consensus.cc:729] T 00000000000000000000000000000000 P e689031926364272be42e55905f4223d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e689031926364272be42e55905f4223d, State: Initialized, Role: FOLLOWER
I20260925 04:39:12.810868 5711 consensus_queue.cc:260] T 00000000000000000000000000000000 P e689031926364272be42e55905f4223d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e689031926364272be42e55905f4223d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35043 } }
I20260925 04:39:12.811008 5711 raft_consensus.cc:388] T 00000000000000000000000000000000 P e689031926364272be42e55905f4223d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:39:12.811086 5711 raft_consensus.cc:482] T 00000000000000000000000000000000 P e689031926364272be42e55905f4223d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:39:12.811190 5711 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e689031926364272be42e55905f4223d [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:39:12.812247 5711 raft_consensus.cc:504] T 00000000000000000000000000000000 P e689031926364272be42e55905f4223d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e689031926364272be42e55905f4223d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35043 } }
I20260925 04:39:12.812625 5711 leader_election.cc:302] T 00000000000000000000000000000000 P e689031926364272be42e55905f4223d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e689031926364272be42e55905f4223d; no voters:
I20260925 04:39:12.812956 5711 leader_election.cc:288] T 00000000000000000000000000000000 P e689031926364272be42e55905f4223d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:39:12.813089 5716 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e689031926364272be42e55905f4223d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:39:12.813381 5716 raft_consensus.cc:686] T 00000000000000000000000000000000 P e689031926364272be42e55905f4223d [term 1 LEADER]: Becoming Leader. State: Replica: e689031926364272be42e55905f4223d, State: Running, Role: LEADER
I20260925 04:39:12.813755 5716 consensus_queue.cc:237] T 00000000000000000000000000000000 P e689031926364272be42e55905f4223d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e689031926364272be42e55905f4223d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35043 } }
I20260925 04:39:12.814286 5711 sys_catalog.cc:564] T 00000000000000000000000000000000 P e689031926364272be42e55905f4223d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:39:12.815460 5717 sys_catalog.cc:455] T 00000000000000000000000000000000 P e689031926364272be42e55905f4223d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e689031926364272be42e55905f4223d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e689031926364272be42e55905f4223d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35043 } } }
I20260925 04:39:12.815460 5718 sys_catalog.cc:455] T 00000000000000000000000000000000 P e689031926364272be42e55905f4223d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e689031926364272be42e55905f4223d. Latest consensus state: current_term: 1 leader_uuid: "e689031926364272be42e55905f4223d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e689031926364272be42e55905f4223d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35043 } } }
I20260925 04:39:12.815676 5717 sys_catalog.cc:458] T 00000000000000000000000000000000 P e689031926364272be42e55905f4223d [sys.catalog]: This master's current role is: LEADER
I20260925 04:39:12.815681 5718 sys_catalog.cc:458] T 00000000000000000000000000000000 P e689031926364272be42e55905f4223d [sys.catalog]: This master's current role is: LEADER
W20260925 04:39:12.819370 5731 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e689031926364272be42e55905f4223d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 04:39:12.819483 5731 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 04:39:12.819653 5725 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:39:12.820400 5725 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:39:12.826280 5725 catalog_manager.cc:1347] Generated new cluster ID: 50c9f94358ef48fd8978f86bd6a79a84
I20260925 04:39:12.826360 5725 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:39:12.835968 5725 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:39:12.837260 5725 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:39:12.843227 5725 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e689031926364272be42e55905f4223d: Generated new TSK 0
I20260925 04:39:12.844190 5725 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 04:39:12.880091 5741 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260925 04:39:12.880178 5741 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 04:39:12.880970 5714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:12.861199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52906 (local address 127.0.98.62:35043)
0925 04:39:12.861345 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:12.861349 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:12.861389 (+ 40us) server_negotiation.cc:407] Connection header received
0925 04:39:12.861583 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:12.861587 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:12.861654 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:12.861783 (+ 129us) server_negotiation.cc:226] Negotiated authn=JWT
0925 04:39:12.862070 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:12.862922 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:12.863905 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:12.864274 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:12.864377 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:12.864696 (+ 319us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 04:39:12.880503 (+ 15807us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45645/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 04:39:12.880819 (+ 316us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52906: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45645/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":70}
W20260925 04:39:12.881060 5714 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52906: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45645/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260925 04:39:12.881109 5740 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:39:12.861046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35043 (local address 127.0.0.1:52906)
0925 04:39:12.861321 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:12.861333 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:12.861435 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:12.861819 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:12.861822 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:12.861831 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0925 04:39:12.861936 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:12.861940 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:12.863073 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:12.863077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:12.863781 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:12.863790 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:12.864377 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:12.864401 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:12.864485 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:12.864512 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 04:39:12.880761 (+ 16249us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45645/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 04:39:12.880932 (+ 171us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:35043: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45645/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":97,"threads_started":1}
I20260925 04:39:12.881784 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 5644
2026-09-25T04:39:12Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (310 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260925 04:39:12.920656 392 mini_oidc.cc:152] Starting JWKS server
I20260925 04:39:12.921554 392 webserver.cc:460] Webserver started at https://127.0.0.1:45923/ using document root <none> and password file <none>
I20260925 04:39:12.921649 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 04:39:12.921912 392 webserver.cc:460] Webserver started at http://127.0.0.1:34877/ using document root <none> and password file <none>
2026-09-25T04:39:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:39:12Z Disabled control of system clock
I20260925 04:39:12.930840 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790311123478858-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790311123478858-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790311123478858-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39569
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39569
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45923/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 04:39:13.058086 5747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:39:13.058415 5747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:39:13.058475 5747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:39:13.062934 5747 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:39:13.063019 5747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:39:13.063041 5747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:39:13.063062 5747 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:39:13.063082 5747 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 04:39:13.063104 5747 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 04:39:13.063130 5747 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45923/jwks.json
I20260925 04:39:13.068400 5747 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790311123478858-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790311123478858-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39569
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45923/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790311123478858-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:39:13.069605 5747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:39:13.071147 5747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:39:13.080045 5753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:13.080025 5755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:13.080029 5752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:13.080148 5747 server_base.cc:1034] running on GCE node
I20260925 04:39:13.080880 5747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:39:13.082132 5747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:39:13.083318 5747 hybrid_clock.cc:648] HybridClock initialized: now 1790311153083300 us; error 35 us; skew 500 ppm
I20260925 04:39:13.085522 5747 webserver.cc:460] Webserver started at http://127.0.98.62:46725/ using document root <none> and password file <none>
I20260925 04:39:13.086153 5747 fs_manager.cc:359] Metadata directory not provided
I20260925 04:39:13.086226 5747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:39:13.086427 5747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:39:13.088757 5747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790311123478858-392-0/minicluster-data/master-0/data/instance:
uuid: "2e5170a9fc2a48e790c44f06d939837f"
format_stamp: "Formatted at 2026-09-25 04:39:13 on dist-test-slave-dc95"
server_key: "4e7c70057e1e5b290b396eee611ce3bb"
server_key_iv: "fb6ca215b38a64c9b06ff2c0f6b4d5e3"
server_key_version: "encryptionkey@0"
I20260925 04:39:13.089392 5747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790311123478858-392-0/minicluster-data/master-0/wal/instance:
uuid: "2e5170a9fc2a48e790c44f06d939837f"
format_stamp: "Formatted at 2026-09-25 04:39:13 on dist-test-slave-dc95"
server_key: "4e7c70057e1e5b290b396eee611ce3bb"
server_key_iv: "fb6ca215b38a64c9b06ff2c0f6b4d5e3"
server_key_version: "encryptionkey@0"
I20260925 04:39:13.093698 5747 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 04:39:13.096406 5761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:13.097790 5747 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:39:13.097925 5747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790311123478858-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790311123478858-392-0/minicluster-data/master-0/wal
uuid: "2e5170a9fc2a48e790c44f06d939837f"
format_stamp: "Formatted at 2026-09-25 04:39:13 on dist-test-slave-dc95"
server_key: "4e7c70057e1e5b290b396eee611ce3bb"
server_key_iv: "fb6ca215b38a64c9b06ff2c0f6b4d5e3"
server_key_version: "encryptionkey@0"
I20260925 04:39:13.098039 5747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790311123478858-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790311123478858-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790311123478858-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:39:13.113802 5747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:39:13.114555 5747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:39:13.114737 5747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:39:13.122571 5747 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39569
I20260925 04:39:13.122592 5813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39569 every 8 connection(s)
I20260925 04:39:13.123581 5747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
I20260925 04:39:13.126995 5814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:39:13.132179 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 5747
I20260925 04:39:13.132283 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790311123478858-392-0/minicluster-data/master-0/wal/instance
I20260925 04:39:13.132603 392 external_mini_cluster.cc:1428] Setting key 64565a2f54347103211344c44b36c991
I20260925 04:39:13.133268 5814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e5170a9fc2a48e790c44f06d939837f: Bootstrap starting.
I20260925 04:39:13.135725 5814 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e5170a9fc2a48e790c44f06d939837f: Neither blocks nor log segments found. Creating new log.
I20260925 04:39:13.136596 5814 log.cc:824] T 00000000000000000000000000000000 P 2e5170a9fc2a48e790c44f06d939837f: Log is configured to *not* fsync() on all Append() calls
I20260925 04:39:13.138895 5814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e5170a9fc2a48e790c44f06d939837f: No bootstrap required, opened a new log
I20260925 04:39:13.142565 5817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:13.134057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38296 (local address 127.0.98.62:39569)
0925 04:39:13.134734 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:13.134749 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:13.134786 (+ 37us) server_negotiation.cc:407] Connection header received
0925 04:39:13.135473 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:13.135491 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:13.135995 (+ 504us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:13.136320 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:13.136706 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:13.137864 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:13.139156 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:13.139714 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:13.139846 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:13.140694 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:13.140700 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:13.140714 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:39:13.141130 (+ 416us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:13.141189 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:39:13.141210 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:13.141284 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:13.141451 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 04:39:13.141760 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":130,"threads_started":1}
I20260925 04:39:13.143064 5814 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e5170a9fc2a48e790c44f06d939837f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e5170a9fc2a48e790c44f06d939837f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39569 } }
I20260925 04:39:13.143321 5814 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e5170a9fc2a48e790c44f06d939837f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:39:13.143381 5814 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e5170a9fc2a48e790c44f06d939837f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e5170a9fc2a48e790c44f06d939837f, State: Initialized, Role: FOLLOWER
I20260925 04:39:13.144588 5814 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e5170a9fc2a48e790c44f06d939837f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e5170a9fc2a48e790c44f06d939837f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39569 } }
I20260925 04:39:13.144752 5814 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e5170a9fc2a48e790c44f06d939837f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:39:13.144819 5814 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e5170a9fc2a48e790c44f06d939837f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:39:13.144924 5814 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e5170a9fc2a48e790c44f06d939837f [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:39:13.145546 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:39:13.146003 5814 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e5170a9fc2a48e790c44f06d939837f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e5170a9fc2a48e790c44f06d939837f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39569 } }
I20260925 04:39:13.146371 5814 leader_election.cc:302] T 00000000000000000000000000000000 P 2e5170a9fc2a48e790c44f06d939837f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e5170a9fc2a48e790c44f06d939837f; no voters:
I20260925 04:39:13.146733 5814 leader_election.cc:288] T 00000000000000000000000000000000 P 2e5170a9fc2a48e790c44f06d939837f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:39:13.146853 5819 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e5170a9fc2a48e790c44f06d939837f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:39:13.147104 5819 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e5170a9fc2a48e790c44f06d939837f [term 1 LEADER]: Becoming Leader. State: Replica: 2e5170a9fc2a48e790c44f06d939837f, State: Running, Role: LEADER
I20260925 04:39:13.147457 5819 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e5170a9fc2a48e790c44f06d939837f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e5170a9fc2a48e790c44f06d939837f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39569 } }
I20260925 04:39:13.148180 5814 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e5170a9fc2a48e790c44f06d939837f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:39:13.149246 5820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e5170a9fc2a48e790c44f06d939837f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e5170a9fc2a48e790c44f06d939837f. Latest consensus state: current_term: 1 leader_uuid: "2e5170a9fc2a48e790c44f06d939837f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e5170a9fc2a48e790c44f06d939837f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39569 } } }
I20260925 04:39:13.149390 5820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e5170a9fc2a48e790c44f06d939837f [sys.catalog]: This master's current role is: LEADER
I20260925 04:39:13.149752 5828 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:39:13.149632 5821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e5170a9fc2a48e790c44f06d939837f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e5170a9fc2a48e790c44f06d939837f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e5170a9fc2a48e790c44f06d939837f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39569 } } }
I20260925 04:39:13.150508 5821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e5170a9fc2a48e790c44f06d939837f [sys.catalog]: This master's current role is: LEADER
I20260925 04:39:13.152899 5828 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:39:13.158998 5828 catalog_manager.cc:1347] Generated new cluster ID: 2aaa198123cd4461b70b71a981af86c3
I20260925 04:39:13.159104 5828 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:39:13.171342 5828 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:39:13.172751 5828 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:39:13.181054 5828 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e5170a9fc2a48e790c44f06d939837f: Generated new TSK 0
I20260925 04:39:13.181882 5828 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:39:13.203191 5817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:13.198014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38300 (local address 127.0.98.62:39569)
0925 04:39:13.198160 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:13.198165 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:13.198358 (+ 193us) server_negotiation.cc:407] Connection header received
0925 04:39:13.198524 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:13.198527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:13.198583 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:13.198694 (+ 111us) server_negotiation.cc:226] Negotiated authn=JWT
0925 04:39:13.198919 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:13.199455 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:13.200703 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:13.200924 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:13.201002 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:13.201357 (+ 355us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 04:39:13.202798 (+ 1441us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45923/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 04:39:13.203044 (+ 246us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38300: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45923/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":53}
W20260925 04:39:13.203264 5817 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38300: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45923/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260925 04:39:13.205258 5843 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:39:13.197897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39569 (local address 127.0.0.1:38300)
0925 04:39:13.198244 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:13.198259 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:13.198383 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:13.198709 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:13.198712 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:13.198720 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0925 04:39:13.198832 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:13.198839 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:13.199552 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:13.199555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:13.200589 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:13.200599 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:13.201015 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:13.201066 (+ 51us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:13.201173 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:13.201203 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 04:39:13.202957 (+ 1754us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45923/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 04:39:13.205073 (+ 2116us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:39569: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45923/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":261,"mutex_wait_us":1810,"thread_start_us":100,"threads_started":1}
I20260925 04:39:13.205677 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 5747
E20260925 04:39:13.216154 5844 webserver.cc:533] Webserver: sslize failed:
E20260925 04:39:13.216246 5844 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-25T04:39:13Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (325 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-25T04:39:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:39:13Z Disabled control of system clock
I20260925 04:39:13.253968 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790311123478858-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790311123478858-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790311123478858-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45779
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45779
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790311123478858-392-0/testchainca.cert with env {}
W20260925 04:39:13.367271 5848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:39:13.367581 5848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:39:13.367645 5848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:39:13.372344 5848 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:39:13.372452 5848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:39:13.372468 5848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:39:13.372491 5848 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:39:13.372506 5848 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 04:39:13.372522 5848 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 04:39:13.372540 5848 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260925 04:39:13.378403 5848 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790311123478858-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790311123478858-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45779
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790311123478858-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790311123478858-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:39:13.379549 5848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:39:13.380609 5848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:39:13.390332 5854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:13.390332 5853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:13.390350 5856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:13.390763 5848 server_base.cc:1034] running on GCE node
I20260925 04:39:13.391352 5848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:39:13.392590 5848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:39:13.393772 5848 hybrid_clock.cc:648] HybridClock initialized: now 1790311153393753 us; error 34 us; skew 500 ppm
I20260925 04:39:13.395913 5848 webserver.cc:460] Webserver started at http://127.0.98.62:40749/ using document root <none> and password file <none>
I20260925 04:39:13.396512 5848 fs_manager.cc:359] Metadata directory not provided
I20260925 04:39:13.396577 5848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:39:13.396775 5848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:39:13.398990 5848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790311123478858-392-0/minicluster-data/master-0/data/instance:
uuid: "07537907d10e48698c428aee8ac523ef"
format_stamp: "Formatted at 2026-09-25 04:39:13 on dist-test-slave-dc95"
server_key: "a2e885e11890c05666db5d8a9cc5155b"
server_key_iv: "8a0e8d149cf40b84bd659b5f3f3307ab"
server_key_version: "encryptionkey@0"
I20260925 04:39:13.399561 5848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790311123478858-392-0/minicluster-data/master-0/wal/instance:
uuid: "07537907d10e48698c428aee8ac523ef"
format_stamp: "Formatted at 2026-09-25 04:39:13 on dist-test-slave-dc95"
server_key: "a2e885e11890c05666db5d8a9cc5155b"
server_key_iv: "8a0e8d149cf40b84bd659b5f3f3307ab"
server_key_version: "encryptionkey@0"
I20260925 04:39:13.403400 5848 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 04:39:13.406256 5862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:13.407830 5848 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:39:13.407981 5848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790311123478858-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790311123478858-392-0/minicluster-data/master-0/wal
uuid: "07537907d10e48698c428aee8ac523ef"
format_stamp: "Formatted at 2026-09-25 04:39:13 on dist-test-slave-dc95"
server_key: "a2e885e11890c05666db5d8a9cc5155b"
server_key_iv: "8a0e8d149cf40b84bd659b5f3f3307ab"
server_key_version: "encryptionkey@0"
I20260925 04:39:13.408154 5848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790311123478858-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790311123478858-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790311123478858-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:39:13.421958 5848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:39:13.422724 5848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:39:13.422910 5848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:39:13.430186 5848 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45779
I20260925 04:39:13.430256 5914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45779 every 8 connection(s)
I20260925 04:39:13.431214 5848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
I20260925 04:39:13.434507 5915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:39:13.440200 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 5848
I20260925 04:39:13.440336 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790311123478858-392-0/minicluster-data/master-0/wal/instance
I20260925 04:39:13.440634 392 external_mini_cluster.cc:1428] Setting key 88c2afcb32baea7c4cf177a0b6ef3f71
I20260925 04:39:13.440929 5915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07537907d10e48698c428aee8ac523ef: Bootstrap starting.
I20260925 04:39:13.443580 5915 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07537907d10e48698c428aee8ac523ef: Neither blocks nor log segments found. Creating new log.
I20260925 04:39:13.444409 5915 log.cc:824] T 00000000000000000000000000000000 P 07537907d10e48698c428aee8ac523ef: Log is configured to *not* fsync() on all Append() calls
I20260925 04:39:13.446617 5915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07537907d10e48698c428aee8ac523ef: No bootstrap required, opened a new log
I20260925 04:39:13.450536 5918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:13.442043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41030 (local address 127.0.98.62:45779)
0925 04:39:13.442749 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:13.442767 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:13.442803 (+ 36us) server_negotiation.cc:407] Connection header received
0925 04:39:13.443516 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:13.443535 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:13.444054 (+ 519us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:13.444423 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:13.444913 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:13.446063 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:13.447017 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:13.447531 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:13.447720 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:13.448562 (+ 842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:13.448567 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:13.448585 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:39:13.449025 (+ 440us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:13.449103 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:39:13.449130 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:13.449226 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:13.449391 (+ 165us) server_negotiation.cc:299] Negotiation successful
0925 04:39:13.449696 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":147,"threads_started":1}
I20260925 04:39:13.450526 5915 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07537907d10e48698c428aee8ac523ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07537907d10e48698c428aee8ac523ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45779 } }
I20260925 04:39:13.450867 5915 raft_consensus.cc:374] T 00000000000000000000000000000000 P 07537907d10e48698c428aee8ac523ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:39:13.450920 5915 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07537907d10e48698c428aee8ac523ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07537907d10e48698c428aee8ac523ef, State: Initialized, Role: FOLLOWER
I20260925 04:39:13.451434 5915 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07537907d10e48698c428aee8ac523ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07537907d10e48698c428aee8ac523ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45779 } }
I20260925 04:39:13.451624 5915 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07537907d10e48698c428aee8ac523ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:39:13.451689 5915 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07537907d10e48698c428aee8ac523ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:39:13.451776 5915 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07537907d10e48698c428aee8ac523ef [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:39:13.452883 5915 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07537907d10e48698c428aee8ac523ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07537907d10e48698c428aee8ac523ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45779 } }
I20260925 04:39:13.453276 5915 leader_election.cc:302] T 00000000000000000000000000000000 P 07537907d10e48698c428aee8ac523ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07537907d10e48698c428aee8ac523ef; no voters:
I20260925 04:39:13.453632 5915 leader_election.cc:288] T 00000000000000000000000000000000 P 07537907d10e48698c428aee8ac523ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:39:13.453778 5920 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07537907d10e48698c428aee8ac523ef [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:39:13.453984 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:39:13.454035 5920 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07537907d10e48698c428aee8ac523ef [term 1 LEADER]: Becoming Leader. State: Replica: 07537907d10e48698c428aee8ac523ef, State: Running, Role: LEADER
I20260925 04:39:13.454388 5920 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07537907d10e48698c428aee8ac523ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07537907d10e48698c428aee8ac523ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45779 } }
I20260925 04:39:13.455024 5915 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07537907d10e48698c428aee8ac523ef [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:39:13.456422 5922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07537907d10e48698c428aee8ac523ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07537907d10e48698c428aee8ac523ef. Latest consensus state: current_term: 1 leader_uuid: "07537907d10e48698c428aee8ac523ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07537907d10e48698c428aee8ac523ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45779 } } }
I20260925 04:39:13.456560 5922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07537907d10e48698c428aee8ac523ef [sys.catalog]: This master's current role is: LEADER
I20260925 04:39:13.456914 5921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07537907d10e48698c428aee8ac523ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "07537907d10e48698c428aee8ac523ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07537907d10e48698c428aee8ac523ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45779 } } }
I20260925 04:39:13.457010 5921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07537907d10e48698c428aee8ac523ef [sys.catalog]: This master's current role is: LEADER
W20260925 04:39:13.459980 5935 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 07537907d10e48698c428aee8ac523ef: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 04:39:13.460068 5935 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 04:39:13.460175 5930 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:39:13.460923 5930 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:39:13.467159 5930 catalog_manager.cc:1347] Generated new cluster ID: 8674e71c36c043cfb430148034a14f6b
I20260925 04:39:13.467254 5930 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:39:13.475517 5930 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:39:13.476730 5930 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:39:13.484149 5930 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 07537907d10e48698c428aee8ac523ef: Generated new TSK 0
I20260925 04:39:13.485065 5930 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:39:13.552330 5918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:13.496394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41044 (local address 127.0.98.62:45779)
0925 04:39:13.496533 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:13.496538 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:13.496751 (+ 213us) server_negotiation.cc:407] Connection header received
0925 04:39:13.497030 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:13.497033 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:13.497094 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:13.497200 (+ 106us) server_negotiation.cc:226] Negotiated authn=JWT
0925 04:39:13.497493 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:13.498230 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:13.499513 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:13.499867 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:13.499959 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:13.500360 (+ 401us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 04:39:13.551823 (+ 51463us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 04:39:13.552161 (+ 338us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41044: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":45}
W20260925 04:39:13.552428 5918 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41044: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260925 04:39:13.552438 5944 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:39:13.496268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45779 (local address 127.0.0.1:41044)
0925 04:39:13.496620 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:13.496635 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:13.496773 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:13.497219 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:13.497222 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:13.497231 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0925 04:39:13.497361 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:13.497367 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:13.498348 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:13.498352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:13.499376 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:13.499387 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:13.499973 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:13.500004 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:13.500108 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:13.500143 (+ 35us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 04:39:13.552087 (+ 51944us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 04:39:13.552275 (+ 188us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:45779: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":120,"threads_started":1}
I20260925 04:39:13.553172 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 5848
2026-09-25T04:39:13Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (325 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (128 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (127 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:39:13 dist-test-slave-dc95 krb5kdc[5956](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:39:13 dist-test-slave-dc95 krb5kdc[5956](info): set up 2 sockets
Sep 25 04:39:13 dist-test-slave-dc95 krb5kdc[5956](info): commencing operation
krb5kdc: starting...
W20260925 04:39:14.767900 392 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/krb5kdc/test-user.keytab.
Sep 25 04:39:14 dist-test-slave-dc95 krb5kdc[5956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:39:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:39:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 04:39:14.891935 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34459
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34459
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:39:15.007347 5969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:39:15.007700 5969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:39:15.007756 5969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:39:15.012111 5969 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:39:15.012209 5969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:39:15.012229 5969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:39:15.012246 5969 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:39:15.012260 5969 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:39:15.018113 5969 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34459
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:39:15.019263 5969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:39:15.020444 5969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:39:15.029836 5974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:15.029872 5977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:15.029839 5975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:15.029896 5969 server_base.cc:1034] running on GCE node
I20260925 04:39:15.030719 5969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:39:15.032038 5969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:39:15.033246 5969 hybrid_clock.cc:648] HybridClock initialized: now 1790311155033221 us; error 47 us; skew 500 ppm
Sep 25 04:39:15 dist-test-slave-dc95 krb5kdc[5956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311155, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:39:15.036458 5969 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 04:39:15.037546 5969 webserver.cc:460] Webserver started at http://127.0.98.62:45527/ using document root <none> and password file <none>
I20260925 04:39:15.038180 5969 fs_manager.cc:359] Metadata directory not provided
I20260925 04:39:15.038256 5969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:39:15.038482 5969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:39:15.040764 5969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/minicluster-data/master-0/data/instance:
uuid: "04cf9fa24ef4416fbbf031b4cd04f928"
format_stamp: "Formatted at 2026-09-25 04:39:15 on dist-test-slave-dc95"
server_key: "4e87b1a3604c1376e784ae23034f2bb3"
server_key_iv: "207a4c183ee48ea45e190aabd5f70da6"
server_key_version: "encryptionkey@0"
I20260925 04:39:15.041383 5969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/minicluster-data/master-0/wal/instance:
uuid: "04cf9fa24ef4416fbbf031b4cd04f928"
format_stamp: "Formatted at 2026-09-25 04:39:15 on dist-test-slave-dc95"
server_key: "4e87b1a3604c1376e784ae23034f2bb3"
server_key_iv: "207a4c183ee48ea45e190aabd5f70da6"
server_key_version: "encryptionkey@0"
I20260925 04:39:15.045781 5969 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 04:39:15.048693 5984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:15.050221 5969 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:39:15.050393 5969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/minicluster-data/master-0/wal
uuid: "04cf9fa24ef4416fbbf031b4cd04f928"
format_stamp: "Formatted at 2026-09-25 04:39:15 on dist-test-slave-dc95"
server_key: "4e87b1a3604c1376e784ae23034f2bb3"
server_key_iv: "207a4c183ee48ea45e190aabd5f70da6"
server_key_version: "encryptionkey@0"
I20260925 04:39:15.050536 5969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:39:15.068430 5969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:39:15.072095 5969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:39:15.072301 5969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:39:15.079631 5969 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34459
I20260925 04:39:15.079877 6036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34459 every 8 connection(s)
I20260925 04:39:15.080704 5969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
I20260925 04:39:15.083724 6037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:39:15.088886 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 5969
I20260925 04:39:15.089012 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/minicluster-data/master-0/wal/instance
I20260925 04:39:15.089263 392 external_mini_cluster.cc:1428] Setting key 64ad9b894a66395ccdae840929650199
I20260925 04:39:15.089548 6037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04cf9fa24ef4416fbbf031b4cd04f928: Bootstrap starting.
I20260925 04:39:15.092235 6037 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04cf9fa24ef4416fbbf031b4cd04f928: Neither blocks nor log segments found. Creating new log.
I20260925 04:39:15.093192 6037 log.cc:824] T 00000000000000000000000000000000 P 04cf9fa24ef4416fbbf031b4cd04f928: Log is configured to *not* fsync() on all Append() calls
I20260925 04:39:15.095327 6037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04cf9fa24ef4416fbbf031b4cd04f928: No bootstrap required, opened a new log
Sep 25 04:39:15 dist-test-slave-dc95 krb5kdc[5956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:39:15.099731 6037 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04cf9fa24ef4416fbbf031b4cd04f928 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04cf9fa24ef4416fbbf031b4cd04f928" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34459 } }
I20260925 04:39:15.099995 6037 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04cf9fa24ef4416fbbf031b4cd04f928 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:39:15.100063 6037 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04cf9fa24ef4416fbbf031b4cd04f928 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04cf9fa24ef4416fbbf031b4cd04f928, State: Initialized, Role: FOLLOWER
I20260925 04:39:15.100533 6037 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04cf9fa24ef4416fbbf031b4cd04f928 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04cf9fa24ef4416fbbf031b4cd04f928" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34459 } }
I20260925 04:39:15.100667 6037 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04cf9fa24ef4416fbbf031b4cd04f928 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:39:15.100732 6037 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04cf9fa24ef4416fbbf031b4cd04f928 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:39:15.100832 6037 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04cf9fa24ef4416fbbf031b4cd04f928 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:39:15.101913 6037 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04cf9fa24ef4416fbbf031b4cd04f928 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04cf9fa24ef4416fbbf031b4cd04f928" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34459 } }
I20260925 04:39:15.102275 6037 leader_election.cc:302] T 00000000000000000000000000000000 P 04cf9fa24ef4416fbbf031b4cd04f928 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04cf9fa24ef4416fbbf031b4cd04f928; no voters:
I20260925 04:39:15.102582 6037 leader_election.cc:288] T 00000000000000000000000000000000 P 04cf9fa24ef4416fbbf031b4cd04f928 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:39:15.102690 6042 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04cf9fa24ef4416fbbf031b4cd04f928 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:39:15.102998 6042 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04cf9fa24ef4416fbbf031b4cd04f928 [term 1 LEADER]: Becoming Leader. State: Replica: 04cf9fa24ef4416fbbf031b4cd04f928, State: Running, Role: LEADER
I20260925 04:39:15.103353 6042 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04cf9fa24ef4416fbbf031b4cd04f928 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04cf9fa24ef4416fbbf031b4cd04f928" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34459 } }
I20260925 04:39:15.103914 6037 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04cf9fa24ef4416fbbf031b4cd04f928 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:39:15.104681 6040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:15.090662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42806 (local address 127.0.98.62:34459)
0925 04:39:15.091114 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:15.091131 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:15.091170 (+ 39us) server_negotiation.cc:407] Connection header received
0925 04:39:15.091952 (+ 782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:15.091974 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:15.092343 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:15.092691 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:15.093582 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:15.094647 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:15.095824 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:15.096340 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:15.096480 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:15.099279 (+ 2799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:15.099285 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:15.099300 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:15.099333 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:15.101123 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:15.101608 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:15.101612 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:15.101617 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:15.101735 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:15.102101 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:15.102104 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:15.102105 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:15.102584 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:15.102833 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:15.103336 (+ 503us) server_negotiation.cc:299] Negotiation successful
0925 04:39:15.103683 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":136,"threads_started":1}
I20260925 04:39:15.105067 6043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04cf9fa24ef4416fbbf031b4cd04f928 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04cf9fa24ef4416fbbf031b4cd04f928" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04cf9fa24ef4416fbbf031b4cd04f928" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34459 } } }
I20260925 04:39:15.105182 6043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04cf9fa24ef4416fbbf031b4cd04f928 [sys.catalog]: This master's current role is: LEADER
I20260925 04:39:15.105577 6044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04cf9fa24ef4416fbbf031b4cd04f928 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04cf9fa24ef4416fbbf031b4cd04f928. Latest consensus state: current_term: 1 leader_uuid: "04cf9fa24ef4416fbbf031b4cd04f928" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04cf9fa24ef4416fbbf031b4cd04f928" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34459 } } }
I20260925 04:39:15.105702 6044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04cf9fa24ef4416fbbf031b4cd04f928 [sys.catalog]: This master's current role is: LEADER
I20260925 04:39:15.106107 6051 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:39:15.109242 6051 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:39:15.114948 6051 catalog_manager.cc:1347] Generated new cluster ID: 1be6f4cae4f742f09d6a800a87eab4ee
I20260925 04:39:15.115097 6051 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:39:15.129583 6051 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:39:15.130564 6051 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:39:15.135883 6051 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04cf9fa24ef4416fbbf031b4cd04f928: Generated new TSK 0
I20260925 04:39:15.136616 6051 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 04:39:15.175175 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34459
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:39:15.297901 6063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:39:15.298209 6063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:39:15.298269 6063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:39:15.302769 6063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:39:15.302862 6063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:39:15.302963 6063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 04:39:15.308753 6063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:39:15.309878 6063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:39:15.310931 6063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:39:15.320919 6068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:15.320936 6071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:15.320919 6069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:15.320958 6063 server_base.cc:1034] running on GCE node
I20260925 04:39:15.321682 6063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:39:15.322371 6063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:39:15.323547 6063 hybrid_clock.cc:648] HybridClock initialized: now 1790311155323522 us; error 37 us; skew 500 ppm
Sep 25 04:39:15 dist-test-slave-dc95 krb5kdc[5956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311155, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:39:15.326344 6063 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 04:39:15.327412 6063 webserver.cc:460] Webserver started at http://127.0.98.1:36051/ using document root <none> and password file <none>
I20260925 04:39:15.328002 6063 fs_manager.cc:359] Metadata directory not provided
I20260925 04:39:15.328065 6063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:39:15.328239 6063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:39:15.330389 6063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/minicluster-data/ts-0/data/instance:
uuid: "3b5204b28cde4f6787b02bee3cbd299e"
format_stamp: "Formatted at 2026-09-25 04:39:15 on dist-test-slave-dc95"
server_key: "93322358499159df706b8c4145ab21de"
server_key_iv: "82db8f38e906366a7af7ce33a83c389f"
server_key_version: "encryptionkey@0"
I20260925 04:39:15.330961 6063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3b5204b28cde4f6787b02bee3cbd299e"
format_stamp: "Formatted at 2026-09-25 04:39:15 on dist-test-slave-dc95"
server_key: "93322358499159df706b8c4145ab21de"
server_key_iv: "82db8f38e906366a7af7ce33a83c389f"
server_key_version: "encryptionkey@0"
I20260925 04:39:15.335033 6063 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 04:39:15.337587 6078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:15.338907 6063 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 04:39:15.339030 6063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/minicluster-data/ts-0/wal
uuid: "3b5204b28cde4f6787b02bee3cbd299e"
format_stamp: "Formatted at 2026-09-25 04:39:15 on dist-test-slave-dc95"
server_key: "93322358499159df706b8c4145ab21de"
server_key_iv: "82db8f38e906366a7af7ce33a83c389f"
server_key_version: "encryptionkey@0"
I20260925 04:39:15.339133 6063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:39:15.363226 6063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:39:15.366460 6063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:39:15.366621 6063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:39:15.367223 6063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:39:15.368171 6063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:39:15.368221 6063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:15.368263 6063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:39:15.368276 6063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:15.376852 6063 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34707
I20260925 04:39:15.376874 6191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34707 every 8 connection(s)
I20260925 04:39:15.377725 6063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
I20260925 04:39:15.383055 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 6063
I20260925 04:39:15.383184 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/minicluster-data/ts-0/wal/instance
I20260925 04:39:15.383509 392 external_mini_cluster.cc:1428] Setting key b918097263bb73f55a41a66b6f810bf4
Sep 25 04:39:15 dist-test-slave-dc95 krb5kdc[5956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311155, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 04:39:15.393187 6040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:15.379567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48967 (local address 127.0.98.62:34459)
0925 04:39:15.379732 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:15.379736 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:15.380599 (+ 863us) server_negotiation.cc:407] Connection header received
0925 04:39:15.382012 (+ 1413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:15.382016 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:15.382074 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:15.382148 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:15.383572 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:15.384293 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:15.385819 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:15.386049 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:15.386138 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:15.388689 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:15.388693 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:15.388696 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:15.388728 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:15.390734 (+ 2006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:15.391436 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:15.391440 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:15.391442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:15.391539 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:15.391901 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:15.391905 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:15.391906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:15.392102 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:15.392190 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:15.392884 (+ 694us) server_negotiation.cc:299] Negotiation successful
0925 04:39:15.393021 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 04:39:15.394037 6194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:15.379990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34459 (local address 127.0.98.1:48967)
0925 04:39:15.380447 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:15.380492 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:15.381785 (+ 1293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:15.382320 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:15.382327 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:15.382704 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:15.383385 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:15.383407 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:15.384422 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:15.384426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:15.385672 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:15.385685 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:15.386200 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:15.386204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:15.386412 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:15.387035 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:15.387051 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:15.388483 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:39:15.390860 (+ 2377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:15.390900 (+ 40us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:15.390919 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:15.391321 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:15.391649 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:15.391652 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:15.391655 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:15.391809 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:15.392247 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:39:15.392254 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:39:15.392595 (+ 341us) client_negotiation.cc:769] Sending connection context
0925 04:39:15.392814 (+ 219us) client_negotiation.cc:240] Negotiation successful
0925 04:39:15.393163 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":125,"threads_started":1}
I20260925 04:39:15.395366 6192 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34459
I20260925 04:39:15.395648 6192 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260925 04:39:15.396261 6192 heartbeater.cc:511] Master 127.0.98.62:34459 requested a full tablet report, sending...
I20260925 04:39:15.397820 6001 ts_manager.cc:194] Registered new tserver with Master: 3b5204b28cde4f6787b02bee3cbd299e (127.0.98.1:34707)
I20260925 04:39:15.399230 6001 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48967
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 04:39:15.410894 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34459
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:39:15.524031 6197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:39:15.524322 6197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:39:15.524384 6197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:39:15.528928 6197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:39:15.529009 6197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:39:15.529095 6197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 04:39:15.534803 6197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:39:15.536001 6197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:39:15.537041 6197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:39:15.546932 6202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:15.546926 6203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:15.546943 6205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:15.547047 6197 server_base.cc:1034] running on GCE node
I20260925 04:39:15.547662 6197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:39:15.548458 6197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:39:15.549646 6197 hybrid_clock.cc:648] HybridClock initialized: now 1790311155549579 us; error 76 us; skew 500 ppm
Sep 25 04:39:15 dist-test-slave-dc95 krb5kdc[5956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311155, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:39:15.552501 6197 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 04:39:15.553462 6197 webserver.cc:460] Webserver started at http://127.0.98.2:34285/ using document root <none> and password file <none>
I20260925 04:39:15.554066 6197 fs_manager.cc:359] Metadata directory not provided
I20260925 04:39:15.554136 6197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:39:15.554343 6197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:39:15.556614 6197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/minicluster-data/ts-1/data/instance:
uuid: "79b9bd9b296e4eef939ade4879082bbe"
format_stamp: "Formatted at 2026-09-25 04:39:15 on dist-test-slave-dc95"
server_key: "f7ceb00c88190d292bcadfb445b6965d"
server_key_iv: "cae787b70cc2134940ddc29c4fb985a0"
server_key_version: "encryptionkey@0"
I20260925 04:39:15.557214 6197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/minicluster-data/ts-1/wal/instance:
uuid: "79b9bd9b296e4eef939ade4879082bbe"
format_stamp: "Formatted at 2026-09-25 04:39:15 on dist-test-slave-dc95"
server_key: "f7ceb00c88190d292bcadfb445b6965d"
server_key_iv: "cae787b70cc2134940ddc29c4fb985a0"
server_key_version: "encryptionkey@0"
I20260925 04:39:15.561288 6197 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 04:39:15.564023 6212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:15.565361 6197 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 04:39:15.565510 6197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/minicluster-data/ts-1/wal
uuid: "79b9bd9b296e4eef939ade4879082bbe"
format_stamp: "Formatted at 2026-09-25 04:39:15 on dist-test-slave-dc95"
server_key: "f7ceb00c88190d292bcadfb445b6965d"
server_key_iv: "cae787b70cc2134940ddc29c4fb985a0"
server_key_version: "encryptionkey@0"
I20260925 04:39:15.565625 6197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:39:15.582111 6197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:39:15.585350 6197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:39:15.585539 6197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:39:15.586124 6197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:39:15.586961 6197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:39:15.587009 6197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:15.587060 6197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:39:15.587087 6197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:15.595544 6197 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42785
I20260925 04:39:15.595592 6325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42785 every 8 connection(s)
I20260925 04:39:15.596501 6197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
I20260925 04:39:15.597316 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 6197
I20260925 04:39:15.597425 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/minicluster-data/ts-1/wal/instance
I20260925 04:39:15.597685 392 external_mini_cluster.cc:1428] Setting key dde49a26a233270301e0f59e6f9cbc77
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Sep 25 04:39:15 dist-test-slave-dc95 krb5kdc[5956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311155, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:39:15.613481 6040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:15.598288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48577 (local address 127.0.98.62:34459)
0925 04:39:15.598412 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:15.598417 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:15.599919 (+ 1502us) server_negotiation.cc:407] Connection header received
0925 04:39:15.601297 (+ 1378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:15.601300 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:15.601358 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:15.601432 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:15.602674 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:15.603200 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:15.604586 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:15.604883 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:15.604976 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:15.609612 (+ 4636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:15.609615 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:15.609617 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:15.609647 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:15.611211 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:15.611853 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:15.611857 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:15.611859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:15.611944 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:15.612285 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:15.612289 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:15.612291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:15.612504 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:15.612590 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:15.613161 (+ 571us) server_negotiation.cc:299] Negotiation successful
0925 04:39:15.613298 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 04:39:15.614310 6328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:15.598711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34459 (local address 127.0.98.2:48577)
0925 04:39:15.599736 (+ 1025us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:15.599827 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:15.601076 (+ 1249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:15.601610 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:15.601618 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:15.601994 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:15.602537 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:15.602553 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:15.603326 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:15.603329 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:15.604465 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:15.604476 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:15.605043 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:15.605048 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:15.605227 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:15.605897 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:15.605914 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:15.609382 (+ 3468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:39:15.611364 (+ 1982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:15.611372 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:15.611388 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:15.611744 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:15.612031 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:15.612035 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:15.612037 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:15.612189 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:15.612606 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:39:15.612613 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:39:15.612903 (+ 290us) client_negotiation.cc:769] Sending connection context
0925 04:39:15.613089 (+ 186us) client_negotiation.cc:240] Negotiation successful
0925 04:39:15.613449 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":811,"thread_start_us":137,"threads_started":1}
I20260925 04:39:15.615684 6326 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34459
I20260925 04:39:15.615916 6326 heartbeater.cc:464] Registering TS with master...
I20260925 04:39:15.616418 6326 heartbeater.cc:511] Master 127.0.98.62:34459 requested a full tablet report, sending...
I20260925 04:39:15.617522 6001 ts_manager.cc:194] Registered new tserver with Master: 79b9bd9b296e4eef939ade4879082bbe (127.0.98.2:42785)
I20260925 04:39:15.618129 6001 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48577
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 04:39:15.624912 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34459
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:39:15.739645 6331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:39:15.739987 6331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:39:15.740033 6331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:39:15.744338 6331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:39:15.744410 6331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:39:15.744475 6331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 04:39:15.749933 6331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:39:15.751058 6331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:39:15.752132 6331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:39:15.761904 6336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:15.761955 6331 server_base.cc:1034] running on GCE node
W20260925 04:39:15.761955 6337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:15.762141 6339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:15.762523 6331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:39:15.763171 6331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:39:15.764319 6331 hybrid_clock.cc:648] HybridClock initialized: now 1790311155764288 us; error 43 us; skew 500 ppm
Sep 25 04:39:15 dist-test-slave-dc95 krb5kdc[5956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311155, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:39:15.767203 6331 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 04:39:15.768271 6331 webserver.cc:460] Webserver started at http://127.0.98.3:36299/ using document root <none> and password file <none>
I20260925 04:39:15.768905 6331 fs_manager.cc:359] Metadata directory not provided
I20260925 04:39:15.768978 6331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:39:15.769181 6331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:39:15.771512 6331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/minicluster-data/ts-2/data/instance:
uuid: "2aef61a027c742a6b6e05ef10290958e"
format_stamp: "Formatted at 2026-09-25 04:39:15 on dist-test-slave-dc95"
server_key: "48bdb73cf59dabb4b464b69d710d6d04"
server_key_iv: "502a709158fabd3e8bf3256dfb714d17"
server_key_version: "encryptionkey@0"
I20260925 04:39:15.772192 6331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2aef61a027c742a6b6e05ef10290958e"
format_stamp: "Formatted at 2026-09-25 04:39:15 on dist-test-slave-dc95"
server_key: "48bdb73cf59dabb4b464b69d710d6d04"
server_key_iv: "502a709158fabd3e8bf3256dfb714d17"
server_key_version: "encryptionkey@0"
I20260925 04:39:15.776279 6331 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:39:15.778838 6346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:15.780296 6331 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:39:15.780447 6331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/minicluster-data/ts-2/wal
uuid: "2aef61a027c742a6b6e05ef10290958e"
format_stamp: "Formatted at 2026-09-25 04:39:15 on dist-test-slave-dc95"
server_key: "48bdb73cf59dabb4b464b69d710d6d04"
server_key_iv: "502a709158fabd3e8bf3256dfb714d17"
server_key_version: "encryptionkey@0"
I20260925 04:39:15.780575 6331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:39:15.793848 6331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:39:15.797058 6331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:39:15.797221 6331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:39:15.797775 6331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:39:15.798569 6331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:39:15.798616 6331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:15.798656 6331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:39:15.798676 6331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:15.807667 6331 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36193
I20260925 04:39:15.807722 6459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36193 every 8 connection(s)
I20260925 04:39:15.808651 6331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
I20260925 04:39:15.811420 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 6331
I20260925 04:39:15.811539 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/minicluster-data/ts-2/wal/instance
I20260925 04:39:15.811838 392 external_mini_cluster.cc:1428] Setting key 62979d16dfb7819e9e4e9cb75b27472e
Sep 25 04:39:15 dist-test-slave-dc95 krb5kdc[5956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311155, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:39:15.823805 6040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:15.810746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58219 (local address 127.0.98.62:34459)
0925 04:39:15.810931 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:15.810936 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:15.811897 (+ 961us) server_negotiation.cc:407] Connection header received
0925 04:39:15.813601 (+ 1704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:15.813605 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:15.813662 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:15.813749 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:15.815136 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:15.815740 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:15.816994 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:15.817204 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:15.817278 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:15.819899 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:15.819903 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:15.819905 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:15.819936 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:15.821530 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:15.822130 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:15.822133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:15.822134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:15.822216 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:15.822558 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:15.822561 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:15.822562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:15.822770 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:15.822902 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:15.823500 (+ 598us) server_negotiation.cc:299] Negotiation successful
0925 04:39:15.823673 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 04:39:15.824651 6462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:15.811143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34459 (local address 127.0.98.3:58219)
0925 04:39:15.811700 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:15.811762 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:15.813342 (+ 1580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:15.813952 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:15.813989 (+ 37us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:15.814459 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:15.814989 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:15.815004 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:15.815875 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:15.815879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:15.816885 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:15.816896 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:15.817325 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:15.817328 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:15.817483 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:15.818104 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:15.818122 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:15.819643 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:39:15.821685 (+ 2042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:15.821691 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:15.821707 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:15.822028 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:15.822287 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:15.822289 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:15.822290 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:15.822443 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:15.822904 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:39:15.822909 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:39:15.823225 (+ 316us) client_negotiation.cc:769] Sending connection context
0925 04:39:15.823400 (+ 175us) client_negotiation.cc:240] Negotiation successful
0925 04:39:15.823748 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":133,"threads_started":1}
I20260925 04:39:15.825982 6460 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34459
I20260925 04:39:15.826256 6460 heartbeater.cc:464] Registering TS with master...
I20260925 04:39:15.826879 6460 heartbeater.cc:511] Master 127.0.98.62:34459 requested a full tablet report, sending...
I20260925 04:39:15.827994 6001 ts_manager.cc:194] Registered new tserver with Master: 2aef61a027c742a6b6e05ef10290958e (127.0.98.3:36193)
I20260925 04:39:15.828573 6001 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58219
I20260925 04:39:15.839048 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/krb5kdc/test-admin.keytab.
Sep 25 04:39:15 dist-test-slave-dc95 krb5kdc[5956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:39:15.857151 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260925 04:39:15.861281 6466 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:39:15.858772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55136 (local address 127.0.98.1:34707)
0925 04:39:15.859140 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:15.859156 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:15.859855 (+ 699us) server_negotiation.cc:407] Connection header received
0925 04:39:15.860015 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:15.860023 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:15.860274 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:15.860419 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:15.861100 (+ 681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55136: BlockingRecv error: recv got EOF from 127.0.0.1:55136 (error 108)
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":114,"threads_started":1}
W20260925 04:39:15.861428 392 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/krb5kdc/krb5cc)
Sep 25 04:39:15 dist-test-slave-dc95 krb5kdc[5956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:39:15.862598 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 04:39:15 dist-test-slave-dc95 krb5kdc[5956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:39:15.863559 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 04:39:15 dist-test-slave-dc95 krb5kdc[5956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 04:39:15.873550 6466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:15.864506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55142 (local address 127.0.98.1:34707)
0925 04:39:15.864658 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:15.864662 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:15.864697 (+ 35us) server_negotiation.cc:407] Connection header received
0925 04:39:15.864837 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:15.864839 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:15.864882 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:15.864949 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:15.865465 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:15.866267 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:15.867139 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:15.867508 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:15.867586 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:15.870052 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:15.870062 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:15.870075 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:15.870102 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:15.871708 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:15.872143 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:15.872149 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:15.872150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:15.872227 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:15.872470 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:15.872474 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:15.872475 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:15.872745 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:15.873013 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:15.873208 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 04:39:15.873354 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260925 04:39:15.877502 6466 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:39:15.876448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55144 (local address 127.0.98.1:34707)
0925 04:39:15.876598 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:15.876602 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:15.876707 (+ 105us) server_negotiation.cc:407] Connection header received
0925 04:39:15.876770 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:15.876772 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:15.876813 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:15.876911 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:15.877398 (+ 487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55144: BlockingRecv error: recv got EOF from 127.0.0.1:55144 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260925 04:39:15.877559 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/krb5kdc/krb5cc' of type FILE
Sep 25 04:39:15 dist-test-slave-dc95 krb5kdc[5956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:39:15.878582 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 04:39:15 dist-test-slave-dc95 krb5kdc[5956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:39:15.879590 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 04:39:15 dist-test-slave-dc95 krb5kdc[5956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 04:39:15.888476 6466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:15.880550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55154 (local address 127.0.98.1:34707)
0925 04:39:15.880685 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:15.880688 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:15.880703 (+ 15us) server_negotiation.cc:407] Connection header received
0925 04:39:15.880804 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:15.880807 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:15.880891 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:15.880960 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:15.881438 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:15.881991 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:15.882870 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:15.883095 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:15.883171 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:15.885606 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:15.885609 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:15.885612 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:15.885640 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:15.886887 (+ 1247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:15.887358 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:15.887362 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:15.887364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:15.887450 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:15.887766 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:15.887768 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:15.887769 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:15.887946 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:15.888042 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:15.888218 (+ 176us) server_negotiation.cc:299] Negotiation successful
0925 04:39:15.888319 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260925 04:39:15.891639 6466 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:39:15.890467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55162 (local address 127.0.98.1:34707)
0925 04:39:15.890593 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:15.890597 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:15.890730 (+ 133us) server_negotiation.cc:407] Connection header received
0925 04:39:15.890895 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:15.890897 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:15.890940 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:15.891046 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:15.891489 (+ 443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55162: BlockingRecv error: recv got EOF from 127.0.0.1:55162 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260925 04:39:15.891835 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311123478858-392-0/krb5kdc/krb5cc' of type FILE
Sep 25 04:39:15 dist-test-slave-dc95 krb5kdc[5956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:39:15.892889 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 04:39:15 dist-test-slave-dc95 krb5kdc[5956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:39:15.893810 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 04:39:15 dist-test-slave-dc95 krb5kdc[5956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 04:39:15.902571 6466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:15.894812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55164 (local address 127.0.98.1:34707)
0925 04:39:15.894955 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:15.894960 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:15.895029 (+ 69us) server_negotiation.cc:407] Connection header received
0925 04:39:15.895177 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:15.895181 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:15.895226 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:15.895296 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:15.895839 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:15.896355 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:15.897142 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:15.897352 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:15.897437 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:15.899728 (+ 2291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:15.899731 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:15.899734 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:15.899762 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:15.901053 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:15.901472 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:15.901475 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:15.901476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:15.901550 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:15.901785 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:15.901788 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:15.901790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:15.901965 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:15.902085 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:15.902305 (+ 220us) server_negotiation.cc:299] Negotiation successful
0925 04:39:15.902423 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 04:39:15.904047 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 6063
I20260925 04:39:15.909039 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 6197
I20260925 04:39:15.915839 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 6331
I20260925 04:39:15.920128 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 5969
2026-09-25T04:39:15Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2112 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:39:15 dist-test-slave-dc95 krb5kdc[6480](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:39:15 dist-test-slave-dc95 krb5kdc[6480](info): set up 2 sockets
Sep 25 04:39:15 dist-test-slave-dc95 krb5kdc[6480](info): commencing operation
krb5kdc: starting...
W20260925 04:39:16.875737 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/krb5kdc/test-user.keytab.
Sep 25 04:39:16 dist-test-slave-dc95 krb5kdc[6480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:39:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:39:16Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260925 04:39:16.999495 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41387
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41387
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:39:17.109272 6493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:39:17.109578 6493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:39:17.109635 6493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:39:17.114270 6493 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:39:17.114370 6493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:39:17.114403 6493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:39:17.114425 6493 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:39:17.114445 6493 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:39:17.120239 6493 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:39:17.121389 6493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:39:17.122424 6493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:39:17.131234 6499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:17.131227 6501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:17.131227 6498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:17.131422 6493 server_base.cc:1034] running on GCE node
I20260925 04:39:17.132217 6493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:39:17.133515 6493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:39:17.134716 6493 hybrid_clock.cc:648] HybridClock initialized: now 1790311157134701 us; error 40 us; skew 500 ppm
Sep 25 04:39:17 dist-test-slave-dc95 krb5kdc[6480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311157, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:39:17.142472 6493 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260925 04:39:17.143533 6493 webserver.cc:460] Webserver started at http://127.0.98.62:43427/ using document root <none> and password file <none>
I20260925 04:39:17.144155 6493 fs_manager.cc:359] Metadata directory not provided
I20260925 04:39:17.144232 6493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:39:17.144447 6493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:39:17.146620 6493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/master-0/data/instance:
uuid: "36a07e678247497c977b44bb1cf41c03"
format_stamp: "Formatted at 2026-09-25 04:39:17 on dist-test-slave-dc95"
server_key: "4c816826fec834702ae6fe9b81b84714"
server_key_iv: "19dd2fbc4bba0a37b972a9c29616d8bc"
server_key_version: "encryptionkey@0"
I20260925 04:39:17.147219 6493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/master-0/wal/instance:
uuid: "36a07e678247497c977b44bb1cf41c03"
format_stamp: "Formatted at 2026-09-25 04:39:17 on dist-test-slave-dc95"
server_key: "4c816826fec834702ae6fe9b81b84714"
server_key_iv: "19dd2fbc4bba0a37b972a9c29616d8bc"
server_key_version: "encryptionkey@0"
I20260925 04:39:17.151168 6493 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 04:39:17.153934 6508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:17.155660 6493 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:39:17.155822 6493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/master-0/wal
uuid: "36a07e678247497c977b44bb1cf41c03"
format_stamp: "Formatted at 2026-09-25 04:39:17 on dist-test-slave-dc95"
server_key: "4c816826fec834702ae6fe9b81b84714"
server_key_iv: "19dd2fbc4bba0a37b972a9c29616d8bc"
server_key_version: "encryptionkey@0"
I20260925 04:39:17.155953 6493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:39:17.169349 6493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:39:17.173169 6493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:39:17.173413 6493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:39:17.180737 6493 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41387
I20260925 04:39:17.181129 6560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41387 every 8 connection(s)
I20260925 04:39:17.181782 6493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
I20260925 04:39:17.184876 6561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:39:17.185990 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 6493
I20260925 04:39:17.186131 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/master-0/wal/instance
I20260925 04:39:17.186455 392 external_mini_cluster.cc:1428] Setting key 66ab420cd4e21e5a00ccd4b1ab926d3e
I20260925 04:39:17.191489 6561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36a07e678247497c977b44bb1cf41c03: Bootstrap starting.
I20260925 04:39:17.193993 6561 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36a07e678247497c977b44bb1cf41c03: Neither blocks nor log segments found. Creating new log.
I20260925 04:39:17.194795 6561 log.cc:824] T 00000000000000000000000000000000 P 36a07e678247497c977b44bb1cf41c03: Log is configured to *not* fsync() on all Append() calls
Sep 25 04:39:17 dist-test-slave-dc95 krb5kdc[6480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 04:39:17.197194 6561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36a07e678247497c977b44bb1cf41c03: No bootstrap required, opened a new log
I20260925 04:39:17.201182 6564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:17.188138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53566 (local address 127.0.98.62:41387)
0925 04:39:17.188673 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:17.188690 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:17.188730 (+ 40us) server_negotiation.cc:407] Connection header received
0925 04:39:17.189372 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:17.189391 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:17.189762 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:17.190096 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:17.190933 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:17.192039 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:17.192908 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:17.193416 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:17.193589 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:17.196278 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:17.196284 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:17.196299 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:17.196326 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:17.198137 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:17.198657 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:17.198662 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:17.198667 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:17.198784 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:17.199079 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:17.199082 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:17.199084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:17.199509 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:17.199726 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:17.199990 (+ 264us) server_negotiation.cc:299] Negotiation successful
0925 04:39:17.200272 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":155,"threads_started":1}
I20260925 04:39:17.201246 6561 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36a07e678247497c977b44bb1cf41c03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36a07e678247497c977b44bb1cf41c03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41387 } }
I20260925 04:39:17.201514 6561 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36a07e678247497c977b44bb1cf41c03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:39:17.201563 6561 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36a07e678247497c977b44bb1cf41c03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36a07e678247497c977b44bb1cf41c03, State: Initialized, Role: FOLLOWER
I20260925 04:39:17.201932 6561 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36a07e678247497c977b44bb1cf41c03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36a07e678247497c977b44bb1cf41c03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41387 } }
I20260925 04:39:17.202033 6561 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36a07e678247497c977b44bb1cf41c03 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:39:17.202080 6561 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36a07e678247497c977b44bb1cf41c03 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:39:17.202140 6561 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36a07e678247497c977b44bb1cf41c03 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:39:17.203276 6561 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36a07e678247497c977b44bb1cf41c03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36a07e678247497c977b44bb1cf41c03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41387 } }
I20260925 04:39:17.203644 6561 leader_election.cc:302] T 00000000000000000000000000000000 P 36a07e678247497c977b44bb1cf41c03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36a07e678247497c977b44bb1cf41c03; no voters:
I20260925 04:39:17.203948 6561 leader_election.cc:288] T 00000000000000000000000000000000 P 36a07e678247497c977b44bb1cf41c03 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:39:17.204175 6566 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36a07e678247497c977b44bb1cf41c03 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:39:17.204378 6566 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36a07e678247497c977b44bb1cf41c03 [term 1 LEADER]: Becoming Leader. State: Replica: 36a07e678247497c977b44bb1cf41c03, State: Running, Role: LEADER
I20260925 04:39:17.204660 6566 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36a07e678247497c977b44bb1cf41c03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36a07e678247497c977b44bb1cf41c03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41387 } }
I20260925 04:39:17.205130 6561 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36a07e678247497c977b44bb1cf41c03 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:39:17.206235 6567 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36a07e678247497c977b44bb1cf41c03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36a07e678247497c977b44bb1cf41c03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36a07e678247497c977b44bb1cf41c03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41387 } } }
I20260925 04:39:17.206279 6568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36a07e678247497c977b44bb1cf41c03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36a07e678247497c977b44bb1cf41c03. Latest consensus state: current_term: 1 leader_uuid: "36a07e678247497c977b44bb1cf41c03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36a07e678247497c977b44bb1cf41c03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41387 } } }
I20260925 04:39:17.206369 6568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36a07e678247497c977b44bb1cf41c03 [sys.catalog]: This master's current role is: LEADER
I20260925 04:39:17.206369 6567 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36a07e678247497c977b44bb1cf41c03 [sys.catalog]: This master's current role is: LEADER
W20260925 04:39:17.209942 6581 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 36a07e678247497c977b44bb1cf41c03: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 04:39:17.210019 6581 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 04:39:17.210088 6575 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:39:17.210810 6575 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:39:17.216898 6575 catalog_manager.cc:1347] Generated new cluster ID: 9d4b452dab9342578e9f171d3876ca07
I20260925 04:39:17.216988 6575 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:39:17.232873 6575 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:39:17.234236 6575 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:39:17.251686 6575 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36a07e678247497c977b44bb1cf41c03: Generated new TSK 0
I20260925 04:39:17.252485 6575 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260925 04:39:17.290393 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41387
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:39:17.413823 6587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:39:17.414127 6587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:39:17.414172 6587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:39:17.418401 6587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:39:17.418478 6587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:39:17.418541 6587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 04:39:17.424054 6587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:39:17.425144 6587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:39:17.426163 6587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:39:17.435926 6593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:17.435945 6592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:17.435945 6595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:17.436462 6587 server_base.cc:1034] running on GCE node
I20260925 04:39:17.436896 6587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:39:17.437556 6587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:39:17.438730 6587 hybrid_clock.cc:648] HybridClock initialized: now 1790311157438712 us; error 38 us; skew 500 ppm
Sep 25 04:39:17 dist-test-slave-dc95 krb5kdc[6480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311157, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:39:17.441496 6587 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260925 04:39:17.442390 6587 webserver.cc:460] Webserver started at http://127.0.98.1:35745/ using document root <none> and password file <none>
I20260925 04:39:17.442930 6587 fs_manager.cc:359] Metadata directory not provided
I20260925 04:39:17.442994 6587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:39:17.443177 6587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:39:17.445286 6587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-0/data/instance:
uuid: "8a2be5f60c294266a68a6ea6147f2349"
format_stamp: "Formatted at 2026-09-25 04:39:17 on dist-test-slave-dc95"
server_key: "d2c0d86c301d62366e3312f8a3942f4e"
server_key_iv: "193e5e2cd5123d927870eeac044d31b3"
server_key_version: "encryptionkey@0"
I20260925 04:39:17.445863 6587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8a2be5f60c294266a68a6ea6147f2349"
format_stamp: "Formatted at 2026-09-25 04:39:17 on dist-test-slave-dc95"
server_key: "d2c0d86c301d62366e3312f8a3942f4e"
server_key_iv: "193e5e2cd5123d927870eeac044d31b3"
server_key_version: "encryptionkey@0"
I20260925 04:39:17.449911 6587 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 04:39:17.453048 6602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:17.454496 6587 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 04:39:17.454630 6587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-0/wal
uuid: "8a2be5f60c294266a68a6ea6147f2349"
format_stamp: "Formatted at 2026-09-25 04:39:17 on dist-test-slave-dc95"
server_key: "d2c0d86c301d62366e3312f8a3942f4e"
server_key_iv: "193e5e2cd5123d927870eeac044d31b3"
server_key_version: "encryptionkey@0"
I20260925 04:39:17.454754 6587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:39:17.470026 6587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:39:17.473462 6587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:39:17.473687 6587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:39:17.474313 6587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:39:17.475299 6587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:39:17.475365 6587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:17.475426 6587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:39:17.475457 6587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:17.484408 6587 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38919
I20260925 04:39:17.484447 6715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38919 every 8 connection(s)
I20260925 04:39:17.485385 6587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
I20260925 04:39:17.487800 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 6587
I20260925 04:39:17.487915 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-0/wal/instance
I20260925 04:39:17.488246 392 external_mini_cluster.cc:1428] Setting key f8eaf2461a37481c441938d289be0564
Sep 25 04:39:17 dist-test-slave-dc95 krb5kdc[6480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311157, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260925 04:39:17.500663 6564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:17.487328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44325 (local address 127.0.98.62:41387)
0925 04:39:17.487481 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:17.487485 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:17.488474 (+ 989us) server_negotiation.cc:407] Connection header received
0925 04:39:17.490003 (+ 1529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:17.490005 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:17.490052 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:17.490110 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:17.491807 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:17.492348 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:17.493643 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:17.493886 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:17.493980 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:17.496650 (+ 2670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:17.496653 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:17.496655 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:17.496683 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:17.498024 (+ 1341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:17.498681 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:17.498686 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:17.498687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:17.498782 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:17.499194 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:17.499197 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:17.499199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:17.499404 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:17.499491 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:17.500383 (+ 892us) server_negotiation.cc:299] Negotiation successful
0925 04:39:17.500510 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 04:39:17.501840 6718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:17.487747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41387 (local address 127.0.98.1:44325)
0925 04:39:17.488291 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:17.488354 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:17.489749 (+ 1395us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:17.490322 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:17.490333 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:17.490846 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:17.491586 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:17.491643 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:17.492491 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:17.492495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:17.493505 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:17.493517 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:17.494048 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:17.494052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:17.494249 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:17.494847 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:17.494865 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:17.496437 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:39:17.498195 (+ 1758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:17.498203 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:17.498218 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:17.498501 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:17.498899 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:17.498903 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:17.498905 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:17.499077 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:17.499511 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:39:17.499519 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:39:17.499890 (+ 371us) client_negotiation.cc:769] Sending connection context
0925 04:39:17.500300 (+ 410us) client_negotiation.cc:240] Negotiation successful
0925 04:39:17.500727 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":126,"threads_started":1}
I20260925 04:39:17.503420 6716 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41387
I20260925 04:39:17.503796 6716 heartbeater.cc:464] Registering TS with master...
I20260925 04:39:17.504402 6716 heartbeater.cc:511] Master 127.0.98.62:41387 requested a full tablet report, sending...
I20260925 04:39:17.506096 6525 ts_manager.cc:194] Registered new tserver with Master: 8a2be5f60c294266a68a6ea6147f2349 (127.0.98.1:38919)
I20260925 04:39:17.507318 6525 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44325
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260925 04:39:17.519480 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41387
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:39:17.631714 6721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:39:17.632010 6721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:39:17.632068 6721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:39:17.636197 6721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:39:17.636278 6721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:39:17.636361 6721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 04:39:17.642303 6721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:39:17.643484 6721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:39:17.644613 6721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:39:17.654994 6727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:17.654994 6721 server_base.cc:1034] running on GCE node
W20260925 04:39:17.654994 6726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:17.655021 6729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:17.655738 6721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:39:17.656442 6721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:39:17.657610 6721 hybrid_clock.cc:648] HybridClock initialized: now 1790311157657593 us; error 31 us; skew 500 ppm
Sep 25 04:39:17 dist-test-slave-dc95 krb5kdc[6480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311157, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:39:17.660758 6721 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260925 04:39:17.661645 6721 webserver.cc:460] Webserver started at http://127.0.98.2:41505/ using document root <none> and password file <none>
I20260925 04:39:17.662179 6721 fs_manager.cc:359] Metadata directory not provided
I20260925 04:39:17.662236 6721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:39:17.662402 6721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:39:17.664543 6721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-1/data/instance:
uuid: "12a3672caea9413f8a824f93100ef42c"
format_stamp: "Formatted at 2026-09-25 04:39:17 on dist-test-slave-dc95"
server_key: "f8d0d0d828c6b9b36a16fce1908ca74a"
server_key_iv: "38d83059c625cca995faafb2eed3a4c5"
server_key_version: "encryptionkey@0"
I20260925 04:39:17.665076 6721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-1/wal/instance:
uuid: "12a3672caea9413f8a824f93100ef42c"
format_stamp: "Formatted at 2026-09-25 04:39:17 on dist-test-slave-dc95"
server_key: "f8d0d0d828c6b9b36a16fce1908ca74a"
server_key_iv: "38d83059c625cca995faafb2eed3a4c5"
server_key_version: "encryptionkey@0"
I20260925 04:39:17.669019 6721 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:39:17.671751 6736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:17.673184 6721 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:39:17.673316 6721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-1/wal
uuid: "12a3672caea9413f8a824f93100ef42c"
format_stamp: "Formatted at 2026-09-25 04:39:17 on dist-test-slave-dc95"
server_key: "f8d0d0d828c6b9b36a16fce1908ca74a"
server_key_iv: "38d83059c625cca995faafb2eed3a4c5"
server_key_version: "encryptionkey@0"
I20260925 04:39:17.673448 6721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:39:17.696632 6721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:39:17.699929 6721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:39:17.700117 6721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:39:17.700729 6721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:39:17.701589 6721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:39:17.701639 6721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:17.701694 6721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:39:17.701719 6721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:17.710836 6721 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42843
I20260925 04:39:17.710857 6849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42843 every 8 connection(s)
I20260925 04:39:17.711814 6721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
I20260925 04:39:17.716205 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 6721
I20260925 04:39:17.716333 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-1/wal/instance
I20260925 04:39:17.716609 392 external_mini_cluster.cc:1428] Setting key d2fafaf202ec9399403cd6cbbaa68d60
Sep 25 04:39:17 dist-test-slave-dc95 krb5kdc[6480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311157, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260925 04:39:17.727013 6564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:17.713705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44345 (local address 127.0.98.62:41387)
0925 04:39:17.713829 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:17.713833 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:17.714697 (+ 864us) server_negotiation.cc:407] Connection header received
0925 04:39:17.716172 (+ 1475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:17.716177 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:17.716238 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:17.716337 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:17.718136 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:17.718844 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:17.720157 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:17.720479 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:17.720601 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:17.723388 (+ 2787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:17.723391 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:17.723393 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:17.723419 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:17.724816 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:17.725373 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:17.725376 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:17.725377 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:17.725457 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:17.725812 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:17.725815 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:17.725816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:17.726022 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:17.726109 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:17.726699 (+ 590us) server_negotiation.cc:299] Negotiation successful
0925 04:39:17.726845 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 04:39:17.728021 6852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:17.714057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41387 (local address 127.0.98.2:44345)
0925 04:39:17.714519 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:17.714564 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:17.715873 (+ 1309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:17.716589 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:17.716597 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:17.717232 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:17.717829 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:17.717846 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:17.718977 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:17.718980 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:17.720024 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:17.720034 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:17.720653 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:17.720657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:17.720859 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:17.721537 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:17.721558 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:17.723200 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:39:17.724944 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:17.724951 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:17.724969 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:17.725271 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:17.725541 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:17.725544 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:17.725546 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:17.725686 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:17.726125 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:39:17.726132 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:39:17.726457 (+ 325us) client_negotiation.cc:769] Sending connection context
0925 04:39:17.726676 (+ 219us) client_negotiation.cc:240] Negotiation successful
0925 04:39:17.727035 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":117,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
I20260925 04:39:17.729341 6850 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41387
I20260925 04:39:17.729611 6850 heartbeater.cc:464] Registering TS with master...
I20260925 04:39:17.730227 6850 heartbeater.cc:511] Master 127.0.98.62:41387 requested a full tablet report, sending...
I20260925 04:39:17.731436 6525 ts_manager.cc:194] Registered new tserver with Master: 12a3672caea9413f8a824f93100ef42c (127.0.98.2:42843)
I20260925 04:39:17.732244 6525 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44345
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260925 04:39:17.747733 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41387
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:39:17.861061 6855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:39:17.861450 6855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:39:17.861517 6855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:39:17.866969 6855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:39:17.867046 6855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:39:17.867129 6855 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 04:39:17.872609 6855 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6855
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:39:17.873845 6855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:39:17.874889 6855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:39:17.884553 6863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:17.884585 6860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:17.884609 6861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:17.884764 6855 server_base.cc:1034] running on GCE node
I20260925 04:39:17.885303 6855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:39:17.885948 6855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:39:17.887104 6855 hybrid_clock.cc:648] HybridClock initialized: now 1790311157887076 us; error 42 us; skew 500 ppm
Sep 25 04:39:17 dist-test-slave-dc95 krb5kdc[6480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311157, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:39:17.889817 6855 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260925 04:39:17.890698 6855 webserver.cc:460] Webserver started at http://127.0.98.3:34433/ using document root <none> and password file <none>
I20260925 04:39:17.891227 6855 fs_manager.cc:359] Metadata directory not provided
I20260925 04:39:17.891296 6855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:39:17.891484 6855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:39:17.893712 6855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-2/data/instance:
uuid: "6682fed6b70747e791e1cf424b939759"
format_stamp: "Formatted at 2026-09-25 04:39:17 on dist-test-slave-dc95"
server_key: "b5da8a4da97b1b8cb9a1de1e3c2ae4bf"
server_key_iv: "dc5f9d20f68b52a60965ae0678253f97"
server_key_version: "encryptionkey@0"
I20260925 04:39:17.894263 6855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6682fed6b70747e791e1cf424b939759"
format_stamp: "Formatted at 2026-09-25 04:39:17 on dist-test-slave-dc95"
server_key: "b5da8a4da97b1b8cb9a1de1e3c2ae4bf"
server_key_iv: "dc5f9d20f68b52a60965ae0678253f97"
server_key_version: "encryptionkey@0"
I20260925 04:39:17.898115 6855 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 04:39:17.900561 6870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:17.901815 6855 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:39:17.901919 6855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-2/wal
uuid: "6682fed6b70747e791e1cf424b939759"
format_stamp: "Formatted at 2026-09-25 04:39:17 on dist-test-slave-dc95"
server_key: "b5da8a4da97b1b8cb9a1de1e3c2ae4bf"
server_key_iv: "dc5f9d20f68b52a60965ae0678253f97"
server_key_version: "encryptionkey@0"
I20260925 04:39:17.902033 6855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:39:17.915884 6855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:39:17.918586 6855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:39:17.918758 6855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:39:17.919343 6855 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:39:17.920151 6855 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:39:17.920199 6855 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:17.920238 6855 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:39:17.920279 6855 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:17.928716 6855 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44315
I20260925 04:39:17.928807 6983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44315 every 8 connection(s)
I20260925 04:39:17.929601 6855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
I20260925 04:39:17.934504 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 6855
I20260925 04:39:17.934628 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-2/wal/instance
I20260925 04:39:17.934890 392 external_mini_cluster.cc:1428] Setting key 9ff0a067835131a6938bf4341600ce95
Sep 25 04:39:17 dist-test-slave-dc95 krb5kdc[6480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311157, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 04:39:17.943009 6564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:17.931360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37969 (local address 127.0.98.62:41387)
0925 04:39:17.931532 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:17.931537 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:17.932505 (+ 968us) server_negotiation.cc:407] Connection header received
0925 04:39:17.933839 (+ 1334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:17.933841 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:17.933892 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:17.933968 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:17.935346 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:17.936018 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:17.937325 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:17.937530 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:17.937608 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:17.940099 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:17.940102 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:17.940104 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:17.940126 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:17.941288 (+ 1162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:17.941746 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:17.941749 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:17.941752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:17.941818 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:17.942112 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:17.942114 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:17.942115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:17.942290 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:17.942360 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:17.942802 (+ 442us) server_negotiation.cc:299] Negotiation successful
0925 04:39:17.942891 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 04:39:17.943881 6986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:17.931821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41387 (local address 127.0.98.3:37969)
0925 04:39:17.932331 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:17.932379 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:17.933622 (+ 1243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:17.934150 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:17.934157 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:17.934584 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:17.935167 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:17.935187 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:17.936164 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:17.936168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:17.937225 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:17.937234 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:17.937680 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:17.937683 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:17.937844 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:17.938474 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:17.938492 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:17.939941 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:39:17.941392 (+ 1451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:17.941397 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:17.941409 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:17.941669 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:17.941901 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:17.941904 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:17.941906 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:17.942044 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:17.942364 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:39:17.942369 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:39:17.942598 (+ 229us) client_negotiation.cc:769] Sending connection context
0925 04:39:17.942772 (+ 174us) client_negotiation.cc:240] Negotiation successful
0925 04:39:17.943019 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":118,"threads_started":1}
I20260925 04:39:17.944995 6984 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41387
I20260925 04:39:17.945266 6984 heartbeater.cc:464] Registering TS with master...
I20260925 04:39:17.945751 6984 heartbeater.cc:511] Master 127.0.98.62:41387 requested a full tablet report, sending...
I20260925 04:39:17.946750 6525 ts_manager.cc:194] Registered new tserver with Master: 6682fed6b70747e791e1cf424b939759 (127.0.98.3:44315)
I20260925 04:39:17.947386 6525 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37969
I20260925 04:39:17.948683 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260925 04:39:17.975932 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:46389
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41387,127.0.98.61:46389
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:39:18.087687 6989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:39:18.088002 6989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:39:18.088057 6989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:39:18.092589 6989 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:39:18.092677 6989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:39:18.092705 6989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:39:18.092726 6989 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:39:18.092754 6989 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:39:18.098464 6989 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41387,127.0.98.61:46389
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:46389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
W20260925 04:39:18.098722 6989 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:41387,127.0.98.61:46389'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260925 04:39:18.099689 6989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:39:18.100786 6989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:39:18.110340 6994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:18.110344 6995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:18.110431 6997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:18.110806 6989 server_base.cc:1034] running on GCE node
I20260925 04:39:18.111411 6989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:39:18.112674 6989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:39:18.113855 6989 hybrid_clock.cc:648] HybridClock initialized: now 1790311158113837 us; error 36 us; skew 500 ppm
Sep 25 04:39:18 dist-test-slave-dc95 krb5kdc[6480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311158, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:39:18.117013 6989 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260925 04:39:18.118110 6989 webserver.cc:460] Webserver started at http://127.0.98.61:44425/ using document root <none> and password file <none>
I20260925 04:39:18.118692 6989 fs_manager.cc:359] Metadata directory not provided
I20260925 04:39:18.118762 6989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:39:18.118948 6989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:39:18.121455 6989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/master-1/data/instance:
uuid: "50cabbb535294fa09b1dac97495cde70"
format_stamp: "Formatted at 2026-09-25 04:39:18 on dist-test-slave-dc95"
server_key: "deccaac8dd627b1fb7cd2315eb9f5a50"
server_key_iv: "08f872dce1ee5c5380838792a6e727df"
server_key_version: "encryptionkey@0"
I20260925 04:39:18.122043 6989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/master-1/wal/instance:
uuid: "50cabbb535294fa09b1dac97495cde70"
format_stamp: "Formatted at 2026-09-25 04:39:18 on dist-test-slave-dc95"
server_key: "deccaac8dd627b1fb7cd2315eb9f5a50"
server_key_iv: "08f872dce1ee5c5380838792a6e727df"
server_key_version: "encryptionkey@0"
I20260925 04:39:18.126122 6989 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 04:39:18.128922 7004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:18.130370 6989 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:39:18.130496 6989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/master-1/wal
uuid: "50cabbb535294fa09b1dac97495cde70"
format_stamp: "Formatted at 2026-09-25 04:39:18 on dist-test-slave-dc95"
server_key: "deccaac8dd627b1fb7cd2315eb9f5a50"
server_key_iv: "08f872dce1ee5c5380838792a6e727df"
server_key_version: "encryptionkey@0"
I20260925 04:39:18.130623 6989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:39:18.145161 6989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:39:18.148366 6989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:39:18.148540 6989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:39:18.155222 6989 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:46389
I20260925 04:39:18.155491 7056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:46389 every 8 connection(s)
I20260925 04:39:18.156220 6989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/master-1/data/info.pb
I20260925 04:39:18.159214 7057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:39:18.162319 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 6989
I20260925 04:39:18.163164 7057 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41387 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 04:39:18 dist-test-slave-dc95 krb5kdc[6480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311158, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 25 04:39:18 dist-test-slave-dc95 krb5kdc[6480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260925 04:39:18.175596 6564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:18.165126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53580 (local address 127.0.98.62:41387)
0925 04:39:18.165262 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:18.165267 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:18.165552 (+ 285us) server_negotiation.cc:407] Connection header received
0925 04:39:18.165719 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:18.165722 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:18.165771 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:18.165823 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:18.166717 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.167505 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.168605 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.168828 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.168920 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:18.171500 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:18.171503 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:18.171505 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:18.171534 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:18.173079 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:18.173702 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:18.173706 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:18.173708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:18.173807 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:18.174223 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:18.174227 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:18.174228 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:18.174520 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:18.174613 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:18.175240 (+ 627us) server_negotiation.cc:299] Negotiation successful
0925 04:39:18.175393 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 04:39:18.176601 7060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:18.163846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45036 (local address 127.0.98.61:46389)
0925 04:39:18.164258 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:18.164276 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:18.164319 (+ 43us) server_negotiation.cc:407] Connection header received
0925 04:39:18.165053 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:18.165073 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:18.165451 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:18.165731 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:18.166501 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.167821 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.168615 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.168975 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.169109 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:18.171915 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:18.171921 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:18.171933 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:18.171955 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:18.173631 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:18.174176 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:18.174182 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:18.174184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:18.174282 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:18.174589 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:18.174592 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:18.174594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:18.175013 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:18.175279 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:18.175522 (+ 243us) server_negotiation.cc:299] Negotiation successful
0925 04:39:18.175836 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":268,"thread_start_us":101,"threads_started":1}
I20260925 04:39:18.176599 7061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:18.165030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41387 (local address 127.0.0.1:53580)
0925 04:39:18.165408 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:18.165460 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:18.165595 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:18.165982 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:18.165987 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:18.166328 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:18.166575 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:18.166586 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.167684 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.167688 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:18.168476 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:18.168485 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.168970 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.168973 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:18.169103 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:18.169739 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:18.169754 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:18.171290 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:39:18.173223 (+ 1933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:18.173232 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:18.173242 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:18.173578 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:18.173904 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:18.173907 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:18.173908 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:18.174125 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:18.174656 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:39:18.174663 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:39:18.174985 (+ 322us) client_negotiation.cc:769] Sending connection context
0925 04:39:18.175148 (+ 163us) client_negotiation.cc:240] Negotiation successful
0925 04:39:18.175522 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":2,"thread_start_us":131,"threads_started":1}
I20260925 04:39:18.178013 7057 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46389 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 04:39:18 dist-test-slave-dc95 krb5kdc[6480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311158, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260925 04:39:18.186221 7061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:18.178483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46389 (local address 127.0.0.1:45042)
0925 04:39:18.178708 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:18.178720 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:18.178812 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:18.179037 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:18.179039 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:18.179192 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:18.179288 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:18.179294 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.180062 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.180065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:18.180699 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:18.180709 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.181187 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.181190 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:18.181288 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:18.181810 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:18.181825 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:18.183207 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:39:18.184781 (+ 1574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:18.184784 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:18.184786 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:18.185014 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:18.185228 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:18.185231 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:18.185233 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:18.185313 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:18.185700 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:39:18.185703 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:39:18.185799 (+ 96us) client_negotiation.cc:769] Sending connection context
0925 04:39:18.185904 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 04:39:18.186049 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170}
I20260925 04:39:18.186259 7060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:18.178613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45042 (local address 127.0.98.61:46389)
0925 04:39:18.178717 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:18.178721 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:18.178765 (+ 44us) server_negotiation.cc:407] Connection header received
0925 04:39:18.178898 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:18.178901 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:18.178941 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:18.178999 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:18.179389 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.179953 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.180854 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.181084 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.181163 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:18.183365 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:18.183368 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:18.183370 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:18.183391 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:18.184688 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:18.185094 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:18.185096 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:18.185097 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:18.185160 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:18.185424 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:18.185427 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:18.185428 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:18.185610 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:18.185693 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:18.185955 (+ 262us) server_negotiation.cc:299] Negotiation successful
0925 04:39:18.186107 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":27}
I20260925 04:39:18.189723 7057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50cabbb535294fa09b1dac97495cde70: Bootstrap starting.
I20260925 04:39:18.192070 7057 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50cabbb535294fa09b1dac97495cde70: Neither blocks nor log segments found. Creating new log.
I20260925 04:39:18.192788 7057 log.cc:824] T 00000000000000000000000000000000 P 50cabbb535294fa09b1dac97495cde70: Log is configured to *not* fsync() on all Append() calls
I20260925 04:39:18.194720 7057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50cabbb535294fa09b1dac97495cde70: No bootstrap required, opened a new log
I20260925 04:39:18.197749 7057 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50cabbb535294fa09b1dac97495cde70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a07e678247497c977b44bb1cf41c03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41387 } } peers { permanent_uuid: "50cabbb535294fa09b1dac97495cde70" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46389 } }
I20260925 04:39:18.197966 7057 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50cabbb535294fa09b1dac97495cde70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:39:18.197999 7057 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50cabbb535294fa09b1dac97495cde70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50cabbb535294fa09b1dac97495cde70, State: Initialized, Role: FOLLOWER
I20260925 04:39:18.198459 7057 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50cabbb535294fa09b1dac97495cde70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a07e678247497c977b44bb1cf41c03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41387 } } peers { permanent_uuid: "50cabbb535294fa09b1dac97495cde70" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46389 } }
I20260925 04:39:18.198935 7063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50cabbb535294fa09b1dac97495cde70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a07e678247497c977b44bb1cf41c03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41387 } } peers { permanent_uuid: "50cabbb535294fa09b1dac97495cde70" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46389 } } }
I20260925 04:39:18.199059 7063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50cabbb535294fa09b1dac97495cde70 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:39:18.199687 7057 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50cabbb535294fa09b1dac97495cde70 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 04:39:18.204052 7074 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 50cabbb535294fa09b1dac97495cde70: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 04:39:18.204133 7074 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 04:39:18.206151 6989 master_runner.cc:428] Detected that this master 50cabbb535294fa09b1dac97495cde70 is joining an existing cluster
I20260925 04:39:18.206219 6989 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:46389
I20260925 04:39:18.217931 7060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:18.208380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45058 (local address 127.0.98.61:46389)
0925 04:39:18.208506 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:18.208511 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:18.208818 (+ 307us) server_negotiation.cc:407] Connection header received
0925 04:39:18.208966 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:18.208968 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:18.209002 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:18.209034 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:18.209845 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.210332 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.211283 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.212770 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.212841 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:18.214273 (+ 1432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:18.214277 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:18.214279 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:18.214310 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:18.215867 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:18.216436 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:18.216439 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:18.216441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:18.216510 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:18.216742 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:18.216745 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:18.216796 (+ 51us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:18.217232 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:18.217339 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:18.217625 (+ 286us) server_negotiation.cc:299] Negotiation successful
0925 04:39:18.217744 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 04:39:18.217979 7081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:18.208466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46389 (local address 127.0.0.1:45058)
0925 04:39:18.208738 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:18.208760 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:18.208878 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:18.209136 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:18.209139 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:18.209466 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:18.209734 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:18.209740 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.210418 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.210420 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:18.211175 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:18.211184 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.212958 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.212961 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:18.213050 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:18.213534 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:18.213550 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:18.214134 (+ 584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:39:18.216143 (+ 2009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:18.216146 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:18.216148 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:18.216343 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:18.216589 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:18.216592 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:18.216594 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:18.216668 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:18.217374 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:39:18.217376 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:39:18.217466 (+ 90us) client_negotiation.cc:769] Sending connection context
0925 04:39:18.217554 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 04:39:18.217823 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":163,"spinlock_wait_cycles":27392,"thread_start_us":108,"threads_started":1}
I20260925 04:39:18.218830 7080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:18.208330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41387 (local address 127.0.0.1:53590)
0925 04:39:18.208739 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:18.208753 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:18.208878 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:18.209132 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:18.209134 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:18.209466 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:18.209731 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:18.209740 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.210773 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.210776 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:18.211655 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:18.211665 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.212507 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.212512 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:18.212790 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:18.213891 (+ 1101us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:18.213911 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:18.214368 (+ 457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:39:18.216963 (+ 2595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:18.216966 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:18.216968 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:18.217348 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:18.217640 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:18.217643 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:18.217645 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:18.217731 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:18.218306 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:39:18.218309 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:39:18.218401 (+ 92us) client_negotiation.cc:769] Sending connection context
0925 04:39:18.218495 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 04:39:18.218635 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":41,"spinlock_wait_cycles":26624,"thread_start_us":90,"threads_started":1}
I20260925 04:39:18.218961 6564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:18.208380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53590 (local address 127.0.98.62:41387)
0925 04:39:18.208508 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:18.208513 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:18.208840 (+ 327us) server_negotiation.cc:407] Connection header received
0925 04:39:18.208977 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:18.208980 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:18.209034 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:18.209109 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:18.209856 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.210656 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.211832 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.212323 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.212465 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:18.214817 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:18.214822 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:18.214825 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:18.214857 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:18.216768 (+ 1911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:18.217436 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:18.217439 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:18.217441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:18.217532 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:18.217839 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:18.217842 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:18.217844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:18.218048 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:18.218115 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:18.218566 (+ 451us) server_negotiation.cc:299] Negotiation successful
0925 04:39:18.218819 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Sep 25 04:39:18 dist-test-slave-dc95 krb5kdc[6480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311157, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260925 04:39:18.234051 7060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:18.223894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45064 (local address 127.0.98.61:46389)
0925 04:39:18.224029 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:18.224034 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:18.224349 (+ 315us) server_negotiation.cc:407] Connection header received
0925 04:39:18.224528 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:18.224531 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:18.224578 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:18.224641 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:18.225420 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.225901 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.227160 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.227384 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.227464 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:18.230232 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:18.230235 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:18.230238 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:18.230274 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:18.231966 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:18.232624 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:18.232628 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:18.232630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:18.232730 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:18.233018 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:18.233020 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:18.233022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:18.233265 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:18.233359 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:18.233765 (+ 406us) server_negotiation.cc:299] Negotiation successful
0925 04:39:18.233893 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 04:39:18.234419 7082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:18.223847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46389 (local address 127.0.0.1:45064)
0925 04:39:18.224213 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:18.224269 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:18.224411 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:18.224687 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:18.224693 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:18.225165 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:18.225301 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:18.225309 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.225995 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.225999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:18.227038 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:18.227050 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.227484 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.227488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:18.227620 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:18.228239 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:18.228254 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:18.230089 (+ 1835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:39:18.232114 (+ 2025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:18.232127 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:18.232135 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:18.232483 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:18.232833 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:18.232836 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:18.232838 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:18.232932 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:18.233377 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:39:18.233386 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:39:18.233534 (+ 148us) client_negotiation.cc:769] Sending connection context
0925 04:39:18.233691 (+ 157us) client_negotiation.cc:240] Negotiation successful
0925 04:39:18.233920 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":99,"threads_started":1}
I20260925 04:39:18.236217 6524 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:46389: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "50cabbb535294fa09b1dac97495cde70"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 46389
}
attrs {
promote: true
}
}
dest_uuid: "36a07e678247497c977b44bb1cf41c03"
cas_config_opid_index: -1
I20260925 04:39:18.236931 6524 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36a07e678247497c977b44bb1cf41c03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "36a07e678247497c977b44bb1cf41c03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41387 } } peers { permanent_uuid: "50cabbb535294fa09b1dac97495cde70" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46389 } attrs { promote: true } }
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260925 04:39:18.239208 7084 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 36a07e678247497c977b44bb1cf41c03 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 50cabbb535294fa09b1dac97495cde70 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "36a07e678247497c977b44bb1cf41c03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41387 } } peers { permanent_uuid: "50cabbb535294fa09b1dac97495cde70" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46389 } attrs { promote: true } } }
I20260925 04:39:18.239665 7031 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50cabbb535294fa09b1dac97495cde70 [term 0 FOLLOWER]: Advancing to term 1
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
I20260925 04:39:18.242076 7084 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:46389
I20260925 04:39:18.242278 7031 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 50cabbb535294fa09b1dac97495cde70 [term 1 FOLLOWER]: Refusing update from remote peer 36a07e678247497c977b44bb1cf41c03: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260925 04:39:18.242650 6989 master.cc:537] Master@127.0.98.61:46389 shutting down...
I20260925 04:39:18.243810 7084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36a07e678247497c977b44bb1cf41c03 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "36a07e678247497c977b44bb1cf41c03" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "36a07e678247497c977b44bb1cf41c03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41387 } } peers { permanent_uuid: "50cabbb535294fa09b1dac97495cde70" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46389 } attrs { promote: true } } }
I20260925 04:39:18.244462 7084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36a07e678247497c977b44bb1cf41c03 [sys.catalog]: This master's current role is: LEADER
I20260925 04:39:18.244692 7084 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 36a07e678247497c977b44bb1cf41c03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50cabbb535294fa09b1dac97495cde70" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46389 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 04:39:18.246567 6511 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260925 04:39:18.247000 6989 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 50cabbb535294fa09b1dac97495cde70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:39:18.247298 6989 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 50cabbb535294fa09b1dac97495cde70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:39:18.247422 6989 tablet_replica.cc:330] T 00000000000000000000000000000000 P 50cabbb535294fa09b1dac97495cde70: stopping tablet replica
W20260925 04:39:18.247570 6511 consensus_peers.cc:487] T 00000000000000000000000000000000 P 36a07e678247497c977b44bb1cf41c03 -> Peer 50cabbb535294fa09b1dac97495cde70 (127.0.98.61:46389): Couldn't send request to peer 50cabbb535294fa09b1dac97495cde70. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 04:39:18.262605 6989 master.cc:552] Master@127.0.98.61:46389 shutdown complete.
I20260925 04:39:18.262678 6989 master_runner.cc:305] Clearing existing system tablet
I20260925 04:39:18.262974 6989 fs_manager.cc:359] Metadata directory not provided
I20260925 04:39:18.263031 6989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:39:18.264457 6989 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 04:39:18.265439 7091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:18.265703 6989 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:39:18.265821 6989 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 04:39:18.265862 6989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/master-1/wal
uuid: "50cabbb535294fa09b1dac97495cde70"
format_stamp: "Formatted at 2026-09-25 04:39:18 on dist-test-slave-dc95"
server_key: "deccaac8dd627b1fb7cd2315eb9f5a50"
server_key_iv: "08f872dce1ee5c5380838792a6e727df"
server_key_version: "encryptionkey@0"
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260925 04:39:18.267964 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35759
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41387,127.0.98.61:46389,127.0.98.60:35759
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:39:18.268081 6989 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 50cabbb535294fa09b1dac97495cde70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:39:18.270221 6989 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 50cabbb535294fa09b1dac97495cde70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 04:39:18.270304 6989 log.cc:1196] T 00000000000000000000000000000000 P 50cabbb535294fa09b1dac97495cde70: Deleting WAL directory at /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260925 04:39:18.270533 6989 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 50cabbb535294fa09b1dac97495cde70: Deleting consensus metadata
I20260925 04:39:18.270682 6989 master_runner.cc:315] Copying system tablet from 127.0.98.62:41387
I20260925 04:39:18.271730 6989 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 50cabbb535294fa09b1dac97495cde70: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:41387
I20260925 04:39:18.279122 6564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:18.272362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53600 (local address 127.0.98.62:41387)
0925 04:39:18.272498 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:18.272503 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:18.272658 (+ 155us) server_negotiation.cc:407] Connection header received
0925 04:39:18.272792 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:18.272795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:18.272839 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:18.272900 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:18.273366 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.273882 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.274690 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.274909 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.274996 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:18.276117 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:18.276121 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:18.276123 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:18.276148 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:18.277567 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:18.278019 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:18.278023 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:18.278025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:18.278104 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:18.278341 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:18.278345 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:18.278346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:18.278527 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:18.278609 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:18.278847 (+ 238us) server_negotiation.cc:299] Negotiation successful
0925 04:39:18.278953 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 04:39:18.279141 7098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:18.272276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41387 (local address 127.0.0.1:53600)
0925 04:39:18.272573 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:18.272587 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:18.272697 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:18.272935 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:18.272938 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:18.273151 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:18.273250 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:18.273256 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.273987 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.273990 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:18.274580 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:18.274589 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.275012 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.275016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:18.275105 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:18.275595 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:18.275635 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:18.275982 (+ 347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:39:18.277680 (+ 1698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:18.277683 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:18.277685 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:18.277930 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:18.278181 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:18.278184 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:18.278186 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:18.278264 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:18.278609 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:39:18.278612 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:39:18.278704 (+ 92us) client_negotiation.cc:769] Sending connection context
0925 04:39:18.278786 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 04:39:18.278953 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":88,"threads_started":1}
I20260925 04:39:18.280027 6545 tablet_copy_service.cc:140] P 36a07e678247497c977b44bb1cf41c03: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 50cabbb535294fa09b1dac97495cde70 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:53600)
I20260925 04:39:18.280194 6545 tablet_copy_service.cc:161] P 36a07e678247497c977b44bb1cf41c03: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 50cabbb535294fa09b1dac97495cde70 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:53600: session id = 50cabbb535294fa09b1dac97495cde70-00000000000000000000000000000000
I20260925 04:39:18.281661 6545 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 36a07e678247497c977b44bb1cf41c03: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 04:39:18.283223 6989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 04:39:18.285161 6989 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 50cabbb535294fa09b1dac97495cde70: tablet copy: Starting download of 0 data blocks...
I20260925 04:39:18.285341 6989 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 50cabbb535294fa09b1dac97495cde70: tablet copy: Starting download of 1 WAL segments...
I20260925 04:39:18.286909 6989 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 50cabbb535294fa09b1dac97495cde70: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 04:39:18.288590 6545 tablet_copy_service.cc:342] P 36a07e678247497c977b44bb1cf41c03: Request end of tablet copy session 50cabbb535294fa09b1dac97495cde70-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:53600
I20260925 04:39:18.288707 6545 tablet_copy_service.cc:433] P 36a07e678247497c977b44bb1cf41c03: ending tablet copy session 50cabbb535294fa09b1dac97495cde70-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 50cabbb535294fa09b1dac97495cde70
I20260925 04:39:18.295861 6989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:39:18.297572 7099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:18.297574 7102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:18.297923 7100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:18.299854 6989 server_base.cc:1034] running on GCE node
I20260925 04:39:18.300034 6989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:39:18.300343 6989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:39:18.301497 6989 hybrid_clock.cc:648] HybridClock initialized: now 1790311158301489 us; error 34 us; skew 500 ppm
Sep 25 04:39:18 dist-test-slave-dc95 krb5kdc[6480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311158, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:39:18.302981 6989 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260925 04:39:18.303436 6989 webserver.cc:460] Webserver started at http://127.0.98.61:33337/ using document root <none> and password file <none>
I20260925 04:39:18.303645 6989 fs_manager.cc:359] Metadata directory not provided
I20260925 04:39:18.303689 6989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:39:18.304766 6989 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 04:39:18.307758 7109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:18.307991 6989 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 04:39:18.308065 6989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/master-1/wal
uuid: "50cabbb535294fa09b1dac97495cde70"
format_stamp: "Formatted at 2026-09-25 04:39:18 on dist-test-slave-dc95"
server_key: "deccaac8dd627b1fb7cd2315eb9f5a50"
server_key_iv: "08f872dce1ee5c5380838792a6e727df"
server_key_version: "encryptionkey@0"
I20260925 04:39:18.308161 6989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:39:18.328055 6989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:39:18.330039 6989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:39:18.334694 6989 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:46389
I20260925 04:39:18.334722 7161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:46389 every 8 connection(s)
I20260925 04:39:18.335255 6989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/master-1/data/info.pb
I20260925 04:39:18.336743 7162 sys_catalog.cc:263] Verifying existing consensus state
I20260925 04:39:18.337592 7162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50cabbb535294fa09b1dac97495cde70: Bootstrap starting.
I20260925 04:39:18.346000 7162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50cabbb535294fa09b1dac97495cde70: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 04:39:18.346292 7162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50cabbb535294fa09b1dac97495cde70: Bootstrap complete.
I20260925 04:39:18.346799 7162 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50cabbb535294fa09b1dac97495cde70 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "36a07e678247497c977b44bb1cf41c03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41387 } } peers { permanent_uuid: "50cabbb535294fa09b1dac97495cde70" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46389 } attrs { promote: true } }
I20260925 04:39:18.346911 7162 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50cabbb535294fa09b1dac97495cde70 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 50cabbb535294fa09b1dac97495cde70, State: Initialized, Role: LEARNER
I20260925 04:39:18.347008 7162 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50cabbb535294fa09b1dac97495cde70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "36a07e678247497c977b44bb1cf41c03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41387 } } peers { permanent_uuid: "50cabbb535294fa09b1dac97495cde70" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46389 } attrs { promote: true } }
I20260925 04:39:18.347333 7165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50cabbb535294fa09b1dac97495cde70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "36a07e678247497c977b44bb1cf41c03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41387 } } peers { permanent_uuid: "50cabbb535294fa09b1dac97495cde70" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46389 } attrs { promote: true } } }
I20260925 04:39:18.347467 7162 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50cabbb535294fa09b1dac97495cde70 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:39:18.347491 7165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50cabbb535294fa09b1dac97495cde70 [sys.catalog]: This master's current role is: LEARNER
I20260925 04:39:18.350590 7176 catalog_manager.cc:1259] Loaded cluster ID: 9d4b452dab9342578e9f171d3876ca07
I20260925 04:39:18.350651 7176 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 50cabbb535294fa09b1dac97495cde70: loading cluster ID for follower catalog manager: success
I20260925 04:39:18.352236 7176 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 50cabbb535294fa09b1dac97495cde70: acquiring CA information for follower catalog manager: success
I20260925 04:39:18.353446 7176 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 50cabbb535294fa09b1dac97495cde70: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 04:39:18.417932 7092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:39:18.418232 7092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:39:18.418277 7092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:39:18.422487 7092 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:39:18.422577 7092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:39:18.422595 7092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:39:18.422610 7092 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:39:18.422626 7092 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:39:18.428217 7092 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41387,127.0.98.61:46389,127.0.98.60:35759
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:39:18.429353 7092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:39:18.430406 7092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:39:18.439760 7181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:18.439957 7182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:18.439931 7092 server_base.cc:1034] running on GCE node
W20260925 04:39:18.439778 7184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:18.440665 7092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:39:18.441823 7092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:39:18.442991 7092 hybrid_clock.cc:648] HybridClock initialized: now 1790311158442969 us; error 35 us; skew 500 ppm
Sep 25 04:39:18 dist-test-slave-dc95 krb5kdc[6480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311158, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:39:18.445863 7092 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260925 04:39:18.446767 7092 webserver.cc:460] Webserver started at http://127.0.98.60:40415/ using document root <none> and password file <none>
I20260925 04:39:18.447386 7092 fs_manager.cc:359] Metadata directory not provided
I20260925 04:39:18.447450 7092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:39:18.447664 7092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:39:18.449929 7092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/master-2/data/instance:
uuid: "d2eabe017416488d9f0a65796832ae84"
format_stamp: "Formatted at 2026-09-25 04:39:18 on dist-test-slave-dc95"
server_key: "f5ce92f9fd1803f06cdaea97ae56217b"
server_key_iv: "2659587b3e56e47c13fa4eba591e42e5"
server_key_version: "encryptionkey@0"
I20260925 04:39:18.450467 7092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/master-2/wal/instance:
uuid: "d2eabe017416488d9f0a65796832ae84"
format_stamp: "Formatted at 2026-09-25 04:39:18 on dist-test-slave-dc95"
server_key: "f5ce92f9fd1803f06cdaea97ae56217b"
server_key_iv: "2659587b3e56e47c13fa4eba591e42e5"
server_key_version: "encryptionkey@0"
I20260925 04:39:18.454519 7092 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260925 04:39:18.457259 7191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:18.458630 7092 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260925 04:39:18.458743 7092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/master-2/wal
uuid: "d2eabe017416488d9f0a65796832ae84"
format_stamp: "Formatted at 2026-09-25 04:39:18 on dist-test-slave-dc95"
server_key: "f5ce92f9fd1803f06cdaea97ae56217b"
server_key_iv: "2659587b3e56e47c13fa4eba591e42e5"
server_key_version: "encryptionkey@0"
I20260925 04:39:18.458876 7092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:39:18.479833 7092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:39:18.483088 7092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:39:18.483245 7092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:39:18.490108 7092 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:35759
I20260925 04:39:18.490414 7243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:35759 every 8 connection(s)
I20260925 04:39:18.491148 7092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/master-2/data/info.pb
I20260925 04:39:18.494107 7244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:39:18.495759 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 7092
I20260925 04:39:18.498850 7244 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41387 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 04:39:18 dist-test-slave-dc95 krb5kdc[6480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 25 04:39:18 dist-test-slave-dc95 krb5kdc[6480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311158, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 04:39:18.509881 6564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:18.500764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53602 (local address 127.0.98.62:41387)
0925 04:39:18.500907 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:18.500911 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:18.501116 (+ 205us) server_negotiation.cc:407] Connection header received
0925 04:39:18.501259 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:18.501262 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:18.501308 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:18.501375 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:18.502050 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.502565 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.503461 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.503754 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.503866 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:18.506616 (+ 2750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:18.506620 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:18.506623 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:18.506660 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:18.508037 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:18.508609 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:18.508613 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:18.508615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:18.508703 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:18.508940 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:18.508943 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:18.508944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:18.509118 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:18.509201 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:18.509589 (+ 388us) server_negotiation.cc:299] Negotiation successful
0925 04:39:18.509716 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 04:39:18.509997 6716 heartbeater.cc:503] Master 127.0.98.62:41387 was elected leader, sending a full tablet report...
I20260925 04:39:18.510823 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:18.497484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46782 (local address 127.0.98.60:35759)
0925 04:39:18.497985 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:18.498000 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:18.498036 (+ 36us) server_negotiation.cc:407] Connection header received
0925 04:39:18.498685 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:18.498704 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:18.499053 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:18.499397 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:18.500212 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.501630 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.502468 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.502958 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.503098 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:18.505993 (+ 2895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:18.506005 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:18.506018 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:18.506042 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:18.507707 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:18.508152 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:18.508159 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:18.508165 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:18.508339 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:18.508596 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:18.508599 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:18.508601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:18.509032 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:18.509211 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:18.509482 (+ 271us) server_negotiation.cc:299] Negotiation successful
0925 04:39:18.509908 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":349,"spinlock_wait_cycles":4352,"thread_start_us":166,"threads_started":1}
I20260925 04:39:18.510819 7247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:18.500594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41387 (local address 127.0.0.1:53602)
0925 04:39:18.501001 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:18.501042 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:18.501157 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:18.501445 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:18.501450 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:18.501810 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:18.501927 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:18.501933 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.502686 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.502690 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:18.503343 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:18.503352 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.503930 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.503933 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:18.504025 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:18.504559 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:18.504577 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:18.506409 (+ 1832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:39:18.508140 (+ 1731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:18.508149 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:18.508169 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:18.508517 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:18.508783 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:18.508786 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:18.508788 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:18.508864 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:18.509213 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:39:18.509221 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:39:18.509371 (+ 150us) client_negotiation.cc:769] Sending connection context
0925 04:39:18.509502 (+ 131us) client_negotiation.cc:240] Negotiation successful
0925 04:39:18.509839 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"spinlock_wait_cycles":2944,"thread_start_us":133,"threads_started":1}
I20260925 04:39:18.512578 7244 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46389 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 04:39:18 dist-test-slave-dc95 krb5kdc[6480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311158, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260925 04:39:18.520663 7247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:18.513034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46389 (local address 127.0.0.1:45066)
0925 04:39:18.513141 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:18.513163 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:18.513234 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:18.513568 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:18.513570 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:18.513706 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:18.513807 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:18.513812 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.514738 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.514741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:18.515351 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:18.515360 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.515911 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.515914 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:18.515998 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:18.516498 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:18.516510 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:18.517722 (+ 1212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:39:18.519313 (+ 1591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:18.519316 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:18.519317 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:18.519495 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:18.519759 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:18.519762 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:18.519763 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:18.519837 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:18.520220 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:39:18.520224 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:39:18.520317 (+ 93us) client_negotiation.cc:769] Sending connection context
0925 04:39:18.520413 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 04:39:18.520529 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260925 04:39:18.520717 7248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:18.513143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45066 (local address 127.0.98.61:46389)
0925 04:39:18.513381 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:18.513384 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:18.513397 (+ 13us) server_negotiation.cc:407] Connection header received
0925 04:39:18.513433 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:18.513435 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:18.513480 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:18.513551 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:18.513939 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.514619 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.515489 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.515775 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.515876 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:18.517867 (+ 1991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:18.517869 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:18.517871 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:18.517895 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:18.519226 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:18.519570 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:18.519573 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:18.519574 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:18.519681 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:18.519919 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:18.519921 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:18.519922 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:18.520109 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:18.520187 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:18.520462 (+ 275us) server_negotiation.cc:299] Negotiation successful
0925 04:39:18.520567 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":87,"threads_started":1}
I20260925 04:39:18.521524 7244 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35759 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 04:39:18 dist-test-slave-dc95 krb5kdc[6480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311158, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260925 04:39:18.529338 7247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:18.522047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35759 (local address 127.0.0.1:46786)
0925 04:39:18.522194 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:18.522209 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:18.522293 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:18.522475 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:18.522478 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:18.522628 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:18.522706 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:18.522711 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.523773 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.523776 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:18.524266 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:18.524272 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.524672 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.524676 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:18.524741 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:18.525180 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:18.525191 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:18.526404 (+ 1213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:39:18.528017 (+ 1613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:18.528020 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:18.528022 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:18.528201 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:18.528441 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:18.528444 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:18.528446 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:18.528512 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:18.528851 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:39:18.528854 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:39:18.528934 (+ 80us) client_negotiation.cc:769] Sending connection context
0925 04:39:18.529016 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 04:39:18.529167 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260925 04:39:18.529429 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:18.522190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46786 (local address 127.0.98.60:35759)
0925 04:39:18.522312 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:18.522316 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:18.522331 (+ 15us) server_negotiation.cc:407] Connection header received
0925 04:39:18.522358 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:18.522361 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:18.522395 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:18.522453 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:18.522805 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.523678 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.524358 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.524576 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.524658 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:18.526535 (+ 1877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:18.526538 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:18.526540 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:18.526561 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:18.527915 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:18.528296 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:18.528299 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:18.528300 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:18.528364 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:18.528585 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:18.528588 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:18.528590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:18.528768 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:18.528850 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:18.529069 (+ 219us) server_negotiation.cc:299] Negotiation successful
0925 04:39:18.529286 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":66}
I20260925 04:39:18.532451 7244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2eabe017416488d9f0a65796832ae84: Bootstrap starting.
I20260925 04:39:18.534963 7244 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2eabe017416488d9f0a65796832ae84: Neither blocks nor log segments found. Creating new log.
I20260925 04:39:18.535778 7244 log.cc:824] T 00000000000000000000000000000000 P d2eabe017416488d9f0a65796832ae84: Log is configured to *not* fsync() on all Append() calls
I20260925 04:39:18.537845 7244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2eabe017416488d9f0a65796832ae84: No bootstrap required, opened a new log
I20260925 04:39:18.541028 7244 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2eabe017416488d9f0a65796832ae84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a07e678247497c977b44bb1cf41c03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41387 } } peers { permanent_uuid: "50cabbb535294fa09b1dac97495cde70" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46389 } } peers { permanent_uuid: "d2eabe017416488d9f0a65796832ae84" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35759 } }
I20260925 04:39:18.541258 7244 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2eabe017416488d9f0a65796832ae84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:39:18.541292 7244 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2eabe017416488d9f0a65796832ae84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2eabe017416488d9f0a65796832ae84, State: Initialized, Role: FOLLOWER
I20260925 04:39:18.541695 7244 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2eabe017416488d9f0a65796832ae84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a07e678247497c977b44bb1cf41c03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41387 } } peers { permanent_uuid: "50cabbb535294fa09b1dac97495cde70" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46389 } } peers { permanent_uuid: "d2eabe017416488d9f0a65796832ae84" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35759 } }
I20260925 04:39:18.542171 7250 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2eabe017416488d9f0a65796832ae84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a07e678247497c977b44bb1cf41c03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41387 } } peers { permanent_uuid: "50cabbb535294fa09b1dac97495cde70" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46389 } } peers { permanent_uuid: "d2eabe017416488d9f0a65796832ae84" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35759 } } }
I20260925 04:39:18.542315 7250 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2eabe017416488d9f0a65796832ae84 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:39:18.542986 7244 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2eabe017416488d9f0a65796832ae84 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 04:39:18.547637 7261 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d2eabe017416488d9f0a65796832ae84: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 04:39:18.547727 7261 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 04:39:18.550882 7092 master_runner.cc:428] Detected that this master d2eabe017416488d9f0a65796832ae84 is joining an existing cluster
I20260925 04:39:18.550949 7092 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:35759
I20260925 04:39:18.561606 6564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:18.553967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53612 (local address 127.0.98.62:41387)
0925 04:39:18.554222 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:18.554226 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:18.554238 (+ 12us) server_negotiation.cc:407] Connection header received
0925 04:39:18.554284 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:18.554286 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:18.554324 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:18.554397 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:18.554941 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.555466 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.556299 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.556505 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.556590 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:18.558261 (+ 1671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:18.558272 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:18.558274 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:18.558298 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:18.559834 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:18.560285 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:18.560288 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:18.560290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:18.560378 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:18.560668 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:18.560672 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:18.560673 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:18.560898 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:18.561000 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:18.561306 (+ 306us) server_negotiation.cc:299] Negotiation successful
0925 04:39:18.561437 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183}
I20260925 04:39:18.562106 7248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:18.553843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45076 (local address 127.0.98.61:46389)
0925 04:39:18.554102 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:18.554107 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:18.554122 (+ 15us) server_negotiation.cc:407] Connection header received
0925 04:39:18.554197 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:18.554201 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:18.554254 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:18.554306 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:18.555049 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.555905 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.557086 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.557447 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.557521 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:18.558824 (+ 1303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:18.558828 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:18.558831 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:18.558858 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:18.560403 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:18.560860 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:18.560864 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:18.560866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:18.560943 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:18.561215 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:18.561219 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:18.561221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:18.561421 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:18.561490 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:18.561839 (+ 349us) server_negotiation.cc:299] Negotiation successful
0925 04:39:18.561965 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181}
I20260925 04:39:18.562579 7268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:18.553403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46389 (local address 127.0.0.1:45076)
0925 04:39:18.554026 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:18.554038 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:18.554108 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:18.554451 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:18.554454 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:18.554733 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:18.554910 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:18.554918 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.556041 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.556045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:18.556952 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:18.556962 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.557684 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.557687 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:18.557779 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:18.558245 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:18.558259 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:18.558678 (+ 419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:39:18.560539 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:18.560542 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:18.560544 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:18.560787 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:18.561034 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:18.561036 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:18.561038 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:18.561115 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:18.561557 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:39:18.561560 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:39:18.561654 (+ 94us) client_negotiation.cc:769] Sending connection context
0925 04:39:18.562269 (+ 615us) client_negotiation.cc:240] Negotiation successful
0925 04:39:18.562395 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"mutex_wait_us":46,"spinlock_wait_cycles":2816,"thread_start_us":85,"threads_started":1}
I20260925 04:39:18.563078 7267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:18.553403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41387 (local address 127.0.0.1:53612)
0925 04:39:18.553723 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:18.553738 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:18.553848 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:18.554429 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:18.554432 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:18.554683 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:18.554824 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:18.554830 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.555562 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.555566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:18.556182 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:18.556191 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.556611 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.556614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:18.556692 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:18.557539 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:18.557550 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:18.557888 (+ 338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:39:18.559944 (+ 2056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:18.559948 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:18.559950 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:18.560189 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:18.560468 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:18.560470 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:18.560473 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:18.560578 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:18.561015 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:39:18.561018 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:39:18.561114 (+ 96us) client_negotiation.cc:769] Sending connection context
0925 04:39:18.562099 (+ 985us) client_negotiation.cc:240] Negotiation successful
0925 04:39:18.562489 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":2,"spinlock_wait_cycles":2432,"thread_start_us":108,"threads_started":1}
I20260925 04:39:18.564110 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 6587
I20260925 04:39:18.571101 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:38919
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:41387
--webserver_port=35745
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:41387,127.0.98.61:46389
--tserver_master_addrs=127.0.98.62:41387,127.0.98.61:46389,127.0.98.60:35759 with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:39:18.572011 7269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:18.553845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35759 (local address 127.0.0.1:46788)
0925 04:39:18.560890 (+ 7045us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:18.560905 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:18.561005 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:18.561277 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:18.561280 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:18.561472 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:18.561583 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:18.561590 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.562545 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.562548 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:18.563448 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:18.563457 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.564127 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.564130 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:18.564755 (+ 625us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:18.565217 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:18.565230 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:18.565622 (+ 392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:39:18.567230 (+ 1608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:18.567233 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:18.567235 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:18.567478 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:18.570708 (+ 3230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:18.570712 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:18.570714 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:18.570835 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:18.571408 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:39:18.571410 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:39:18.571497 (+ 87us) client_negotiation.cc:769] Sending connection context
0925 04:39:18.571640 (+ 143us) client_negotiation.cc:240] Negotiation successful
0925 04:39:18.571821 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6952,"thread_start_us":93,"threads_started":1}
I20260925 04:39:18.572106 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:18.554112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46788 (local address 127.0.98.60:35759)
0925 04:39:18.554194 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:18.554198 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:18.560978 (+ 6780us) server_negotiation.cc:407] Connection header received
0925 04:39:18.561076 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:18.561079 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:18.561116 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:18.561176 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:18.561700 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.562436 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.563561 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.563926 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.564004 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:18.565756 (+ 1752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:18.565759 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:18.565762 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:18.565782 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:18.567123 (+ 1341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:18.567563 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:18.567565 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:18.567566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:18.567662 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:18.570963 (+ 3301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:18.570968 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:18.570970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:18.571225 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:18.571375 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:18.571726 (+ 351us) server_negotiation.cc:299] Negotiation successful
0925 04:39:18.571921 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":32}
Sep 25 04:39:18 dist-test-slave-dc95 krb5kdc[6480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311157, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260925 04:39:18.605393 7082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:18.597845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35759 (local address 127.0.0.1:55130)
0925 04:39:18.598022 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:18.598035 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:18.598128 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:18.598553 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:18.598557 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:18.598768 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:18.598870 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:18.598876 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.599595 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.599599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:18.600181 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:18.600190 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.600648 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.600652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:18.600761 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:18.601188 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:18.601203 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:18.602400 (+ 1197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:39:18.603934 (+ 1534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:18.603937 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:18.603940 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:18.604140 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:18.604394 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:18.604397 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:18.604399 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:18.604479 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:18.604838 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:39:18.604862 (+ 24us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:39:18.604966 (+ 104us) client_negotiation.cc:769] Sending connection context
0925 04:39:18.605057 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 04:39:18.605228 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112}
I20260925 04:39:18.605393 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:18.597904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55130 (local address 127.0.98.60:35759)
0925 04:39:18.598023 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:18.598028 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:18.598079 (+ 51us) server_negotiation.cc:407] Connection header received
0925 04:39:18.598326 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:18.598330 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:18.598399 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:18.598498 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:18.598980 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.599494 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.600291 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.600544 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.600632 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:18.602529 (+ 1897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:18.602532 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:18.602534 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:18.602556 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:18.603830 (+ 1274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:18.604236 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:18.604239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:18.604240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:18.604313 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:18.604557 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:18.604560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:18.604562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:18.604751 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:18.604832 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:18.605120 (+ 288us) server_negotiation.cc:299] Negotiation successful
0925 04:39:18.605226 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 04:39:18.606694 6524 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:35759: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "d2eabe017416488d9f0a65796832ae84"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 35759
}
attrs {
promote: true
}
}
dest_uuid: "36a07e678247497c977b44bb1cf41c03"
cas_config_opid_index: 5
I20260925 04:39:18.607276 6524 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36a07e678247497c977b44bb1cf41c03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "36a07e678247497c977b44bb1cf41c03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41387 } } peers { permanent_uuid: "50cabbb535294fa09b1dac97495cde70" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46389 } attrs { promote: true } } peers { permanent_uuid: "d2eabe017416488d9f0a65796832ae84" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35759 } attrs { promote: true } }
I20260925 04:39:18.608302 7085 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 36a07e678247497c977b44bb1cf41c03 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d2eabe017416488d9f0a65796832ae84 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "36a07e678247497c977b44bb1cf41c03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41387 } } peers { permanent_uuid: "50cabbb535294fa09b1dac97495cde70" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46389 } attrs { promote: true } } peers { permanent_uuid: "d2eabe017416488d9f0a65796832ae84" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35759 } attrs { promote: true } } }
I20260925 04:39:18.609520 7085 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:35759
I20260925 04:39:18.609701 7085 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36a07e678247497c977b44bb1cf41c03 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "36a07e678247497c977b44bb1cf41c03" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "36a07e678247497c977b44bb1cf41c03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41387 } } peers { permanent_uuid: "50cabbb535294fa09b1dac97495cde70" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46389 } attrs { promote: true } } peers { permanent_uuid: "d2eabe017416488d9f0a65796832ae84" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35759 } attrs { promote: true } } }
I20260925 04:39:18.609802 7085 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36a07e678247497c977b44bb1cf41c03 [sys.catalog]: This master's current role is: LEADER
I20260925 04:39:18.610273 7092 master.cc:537] Master@127.0.98.60:35759 shutting down...
W20260925 04:39:18.611999 6511 consensus_peers.cc:487] T 00000000000000000000000000000000 P 36a07e678247497c977b44bb1cf41c03 -> Peer d2eabe017416488d9f0a65796832ae84 (127.0.98.60:35759): Couldn't send request to peer d2eabe017416488d9f0a65796832ae84. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 04:39:18.614977 7092 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d2eabe017416488d9f0a65796832ae84 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260925 04:39:18.615327 7092 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d2eabe017416488d9f0a65796832ae84 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260925 04:39:18.615447 7092 tablet_replica.cc:330] T 00000000000000000000000000000000 P d2eabe017416488d9f0a65796832ae84: stopping tablet replica
I20260925 04:39:18.619221 7248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:18.610580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36210 (local address 127.0.98.61:46389)
0925 04:39:18.610705 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:18.610708 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:18.614430 (+ 3722us) server_negotiation.cc:407] Connection header received
0925 04:39:18.614679 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:18.614683 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:18.614745 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:18.614829 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:39:18.615349 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.616116 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.617782 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.618391 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.618465 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:18.618629 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:18.619005 (+ 376us) server_negotiation.cc:299] Negotiation successful
0925 04:39:18.619087 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 04:39:18.619768 7082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:18.610360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46389 (local address 127.0.0.1:36210)
0925 04:39:18.613923 (+ 3563us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:18.614346 (+ 423us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:18.614530 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:18.614975 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:18.614977 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:18.614997 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:39:18.615210 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:18.615221 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.616451 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.616454 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:18.617637 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:18.617649 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.618738 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.618742 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:18.618831 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:18.618888 (+ 57us) client_negotiation.cc:769] Sending connection context
0925 04:39:18.619527 (+ 639us) client_negotiation.cc:240] Negotiation successful
0925 04:39:18.619631 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3465}
I20260925 04:39:18.620599 7136 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 50cabbb535294fa09b1dac97495cde70 [term 1 LEARNER]: Refusing update from remote peer 36a07e678247497c977b44bb1cf41c03: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 04:39:18.620751 7092 master.cc:552] Master@127.0.98.60:35759 shutdown complete.
I20260925 04:39:18.620801 7092 master_runner.cc:305] Clearing existing system tablet
I20260925 04:39:18.621030 7092 fs_manager.cc:359] Metadata directory not provided
I20260925 04:39:18.621088 7092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:39:18.621116 7086 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 36a07e678247497c977b44bb1cf41c03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50cabbb535294fa09b1dac97495cde70" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46389 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 04:39:18.623106 7136 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 50cabbb535294fa09b1dac97495cde70 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d2eabe017416488d9f0a65796832ae84 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "36a07e678247497c977b44bb1cf41c03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41387 } } peers { permanent_uuid: "50cabbb535294fa09b1dac97495cde70" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46389 } attrs { promote: true } } peers { permanent_uuid: "d2eabe017416488d9f0a65796832ae84" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35759 } attrs { promote: true } } }
I20260925 04:39:18.623914 7092 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:39:18.625016 7274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50cabbb535294fa09b1dac97495cde70 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "36a07e678247497c977b44bb1cf41c03" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "36a07e678247497c977b44bb1cf41c03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41387 } } peers { permanent_uuid: "50cabbb535294fa09b1dac97495cde70" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46389 } attrs { promote: true } } peers { permanent_uuid: "d2eabe017416488d9f0a65796832ae84" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35759 } attrs { promote: true } } }
I20260925 04:39:18.625128 7274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50cabbb535294fa09b1dac97495cde70 [sys.catalog]: This master's current role is: LEARNER
I20260925 04:39:18.625414 7165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50cabbb535294fa09b1dac97495cde70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36a07e678247497c977b44bb1cf41c03. Latest consensus state: current_term: 1 leader_uuid: "36a07e678247497c977b44bb1cf41c03" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "36a07e678247497c977b44bb1cf41c03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41387 } } peers { permanent_uuid: "50cabbb535294fa09b1dac97495cde70" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46389 } attrs { promote: true } } peers { permanent_uuid: "d2eabe017416488d9f0a65796832ae84" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35759 } attrs { promote: true } } }
I20260925 04:39:18.625545 7165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50cabbb535294fa09b1dac97495cde70 [sys.catalog]: This master's current role is: LEARNER
I20260925 04:39:18.626533 7084 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 36a07e678247497c977b44bb1cf41c03: attempting to promote NON_VOTER 50cabbb535294fa09b1dac97495cde70 to VOTER
I20260925 04:39:18.626925 7084 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36a07e678247497c977b44bb1cf41c03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "36a07e678247497c977b44bb1cf41c03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41387 } } peers { permanent_uuid: "50cabbb535294fa09b1dac97495cde70" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46389 } attrs { promote: false } } peers { permanent_uuid: "d2eabe017416488d9f0a65796832ae84" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35759 } attrs { promote: true } }
I20260925 04:39:18.628190 7082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:18.627896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35759 (local address 127.0.0.1:55144)
0925 04:39:18.628015 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:18.628094 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 04:39:18.628368 7136 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 50cabbb535294fa09b1dac97495cde70 [term 1 LEARNER]: Refusing update from remote peer 36a07e678247497c977b44bb1cf41c03: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260925 04:39:18.628870 7086 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 36a07e678247497c977b44bb1cf41c03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50cabbb535294fa09b1dac97495cde70" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46389 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260925 04:39:18.629463 7082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:18.629232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35759 (local address 127.0.0.1:55160)
0925 04:39:18.629340 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:18.629393 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 04:39:18.629976 6511 consensus_peers.cc:487] T 00000000000000000000000000000000 P 36a07e678247497c977b44bb1cf41c03 -> Peer d2eabe017416488d9f0a65796832ae84 (127.0.98.60:35759): Couldn't send request to peer d2eabe017416488d9f0a65796832ae84. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35759: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 04:39:18.630587 7085 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 36a07e678247497c977b44bb1cf41c03 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 50cabbb535294fa09b1dac97495cde70 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "36a07e678247497c977b44bb1cf41c03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41387 } } peers { permanent_uuid: "50cabbb535294fa09b1dac97495cde70" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46389 } attrs { promote: false } } peers { permanent_uuid: "d2eabe017416488d9f0a65796832ae84" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35759 } attrs { promote: true } } }
I20260925 04:39:18.631804 7273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:18.631835 7085 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36a07e678247497c977b44bb1cf41c03 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "36a07e678247497c977b44bb1cf41c03" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "36a07e678247497c977b44bb1cf41c03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41387 } } peers { permanent_uuid: "50cabbb535294fa09b1dac97495cde70" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46389 } attrs { promote: false } } peers { permanent_uuid: "d2eabe017416488d9f0a65796832ae84" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35759 } attrs { promote: true } } }
I20260925 04:39:18.631935 7085 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36a07e678247497c977b44bb1cf41c03 [sys.catalog]: This master's current role is: LEADER
I20260925 04:39:18.632048 7085 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36a07e678247497c977b44bb1cf41c03 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "36a07e678247497c977b44bb1cf41c03" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "36a07e678247497c977b44bb1cf41c03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41387 } } peers { permanent_uuid: "50cabbb535294fa09b1dac97495cde70" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46389 } attrs { promote: false } } peers { permanent_uuid: "d2eabe017416488d9f0a65796832ae84" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35759 } attrs { promote: true } } }
I20260925 04:39:18.632118 7085 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36a07e678247497c977b44bb1cf41c03 [sys.catalog]: This master's current role is: LEADER
I20260925 04:39:18.632182 7092 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:39:18.632311 7092 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.001s sys 0.000s
I20260925 04:39:18.632347 7092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/master-2/wal
uuid: "d2eabe017416488d9f0a65796832ae84"
format_stamp: "Formatted at 2026-09-25 04:39:18 on dist-test-slave-dc95"
server_key: "f5ce92f9fd1803f06cdaea97ae56217b"
server_key_iv: "2659587b3e56e47c13fa4eba591e42e5"
server_key_version: "encryptionkey@0"
I20260925 04:39:18.632834 7136 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 50cabbb535294fa09b1dac97495cde70 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 50cabbb535294fa09b1dac97495cde70 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "36a07e678247497c977b44bb1cf41c03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41387 } } peers { permanent_uuid: "50cabbb535294fa09b1dac97495cde70" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46389 } attrs { promote: false } } peers { permanent_uuid: "d2eabe017416488d9f0a65796832ae84" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35759 } attrs { promote: true } } }
I20260925 04:39:18.634464 7165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50cabbb535294fa09b1dac97495cde70 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "36a07e678247497c977b44bb1cf41c03" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "36a07e678247497c977b44bb1cf41c03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41387 } } peers { permanent_uuid: "50cabbb535294fa09b1dac97495cde70" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46389 } attrs { promote: false } } peers { permanent_uuid: "d2eabe017416488d9f0a65796832ae84" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35759 } attrs { promote: true } } }
I20260925 04:39:18.634564 7165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50cabbb535294fa09b1dac97495cde70 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:39:18.634505 7092 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P d2eabe017416488d9f0a65796832ae84: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:39:18.636619 7092 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P d2eabe017416488d9f0a65796832ae84: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 04:39:18.636693 7092 log.cc:1196] T 00000000000000000000000000000000 P d2eabe017416488d9f0a65796832ae84: Deleting WAL directory at /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260925 04:39:18.636865 7092 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P d2eabe017416488d9f0a65796832ae84: Deleting consensus metadata
I20260925 04:39:18.637019 7092 master_runner.cc:315] Copying system tablet from 127.0.98.62:41387
I20260925 04:39:18.638147 7092 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P d2eabe017416488d9f0a65796832ae84: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:41387
I20260925 04:39:18.650374 7280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:18.639987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41387 (local address 127.0.0.1:49580)
0925 04:39:18.643095 (+ 3108us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:18.643111 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:18.643254 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:18.643538 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:18.643541 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:18.643787 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:18.643911 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:18.643917 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.644837 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.644841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:18.645450 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:18.645460 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.645905 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.645908 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:18.646001 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:18.646505 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:18.646517 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:18.646900 (+ 383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:39:18.648686 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:18.648690 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:18.648692 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:18.648958 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:18.649225 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:18.649228 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:18.649230 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:18.649394 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:18.649783 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:39:18.649786 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:39:18.649887 (+ 101us) client_negotiation.cc:769] Sending connection context
0925 04:39:18.649977 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 04:39:18.650136 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3018,"thread_start_us":98,"threads_started":1}
I20260925 04:39:18.650368 6564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:18.640074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49580 (local address 127.0.98.62:41387)
0925 04:39:18.640213 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:18.640218 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:18.643218 (+ 3000us) server_negotiation.cc:407] Connection header received
0925 04:39:18.643355 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:18.643359 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:18.643415 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:18.643492 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:18.644129 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.644735 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.645572 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.645797 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.645894 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:18.647036 (+ 1142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:18.647040 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:18.647042 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:18.647070 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:18.648583 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:18.649053 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:18.649057 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:18.649059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:18.649143 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:18.649490 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:18.649493 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:18.649495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:18.649692 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:18.649784 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:18.650044 (+ 260us) server_negotiation.cc:299] Negotiation successful
0925 04:39:18.650152 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 04:39:18.651254 6545 tablet_copy_service.cc:140] P 36a07e678247497c977b44bb1cf41c03: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer d2eabe017416488d9f0a65796832ae84 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:49580)
I20260925 04:39:18.651371 6545 tablet_copy_service.cc:161] P 36a07e678247497c977b44bb1cf41c03: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer d2eabe017416488d9f0a65796832ae84 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:49580: session id = d2eabe017416488d9f0a65796832ae84-00000000000000000000000000000000
I20260925 04:39:18.652514 6545 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 36a07e678247497c977b44bb1cf41c03: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 04:39:18.653940 7092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 04:39:18.656085 7092 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P d2eabe017416488d9f0a65796832ae84: tablet copy: Starting download of 0 data blocks...
I20260925 04:39:18.656271 7092 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P d2eabe017416488d9f0a65796832ae84: tablet copy: Starting download of 1 WAL segments...
I20260925 04:39:18.657789 7092 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P d2eabe017416488d9f0a65796832ae84: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 04:39:18.659572 6545 tablet_copy_service.cc:342] P 36a07e678247497c977b44bb1cf41c03: Request end of tablet copy session d2eabe017416488d9f0a65796832ae84-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:49580
I20260925 04:39:18.659698 6545 tablet_copy_service.cc:433] P 36a07e678247497c977b44bb1cf41c03: ending tablet copy session d2eabe017416488d9f0a65796832ae84-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer d2eabe017416488d9f0a65796832ae84
I20260925 04:39:18.666509 7092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:39:18.668205 7281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:18.668349 7092 server_base.cc:1034] running on GCE node
W20260925 04:39:18.668217 7284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:18.671793 7282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:18.672094 7092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:39:18.672438 7092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:39:18.673574 7092 hybrid_clock.cc:648] HybridClock initialized: now 1790311158673563 us; error 33 us; skew 500 ppm
Sep 25 04:39:18 dist-test-slave-dc95 krb5kdc[6480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311158, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:39:18.674953 7092 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260925 04:39:18.675413 7092 webserver.cc:460] Webserver started at http://127.0.98.60:34505/ using document root <none> and password file <none>
I20260925 04:39:18.675595 7092 fs_manager.cc:359] Metadata directory not provided
I20260925 04:39:18.675660 7092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:39:18.676784 7092 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 04:39:18.677781 7291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:18.678005 7092 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 04:39:18.678076 7092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/master-2/wal
uuid: "d2eabe017416488d9f0a65796832ae84"
format_stamp: "Formatted at 2026-09-25 04:39:18 on dist-test-slave-dc95"
server_key: "f5ce92f9fd1803f06cdaea97ae56217b"
server_key_iv: "2659587b3e56e47c13fa4eba591e42e5"
server_key_version: "encryptionkey@0"
I20260925 04:39:18.678159 7092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:39:18.685590 7092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:39:18.687457 7092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:39:18.691118 7092 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:35759
I20260925 04:39:18.691771 7092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/master-2/data/info.pb
I20260925 04:39:18.692775 7343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:35759 every 8 connection(s)
I20260925 04:39:18.694024 7344 sys_catalog.cc:263] Verifying existing consensus state
I20260925 04:39:18.694990 7344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2eabe017416488d9f0a65796832ae84: Bootstrap starting.
I20260925 04:39:18.704414 7344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2eabe017416488d9f0a65796832ae84: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 04:39:18.704710 7344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2eabe017416488d9f0a65796832ae84: Bootstrap complete.
I20260925 04:39:18.705304 7344 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2eabe017416488d9f0a65796832ae84 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "36a07e678247497c977b44bb1cf41c03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41387 } } peers { permanent_uuid: "50cabbb535294fa09b1dac97495cde70" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46389 } attrs { promote: false } } peers { permanent_uuid: "d2eabe017416488d9f0a65796832ae84" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35759 } attrs { promote: true } }
I20260925 04:39:18.705428 7344 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2eabe017416488d9f0a65796832ae84 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d2eabe017416488d9f0a65796832ae84, State: Initialized, Role: LEARNER
I20260925 04:39:18.705528 7344 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2eabe017416488d9f0a65796832ae84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "36a07e678247497c977b44bb1cf41c03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41387 } } peers { permanent_uuid: "50cabbb535294fa09b1dac97495cde70" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46389 } attrs { promote: false } } peers { permanent_uuid: "d2eabe017416488d9f0a65796832ae84" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35759 } attrs { promote: true } }
I20260925 04:39:18.705847 7347 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2eabe017416488d9f0a65796832ae84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "36a07e678247497c977b44bb1cf41c03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41387 } } peers { permanent_uuid: "50cabbb535294fa09b1dac97495cde70" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46389 } attrs { promote: false } } peers { permanent_uuid: "d2eabe017416488d9f0a65796832ae84" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35759 } attrs { promote: true } } }
I20260925 04:39:18.706012 7344 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2eabe017416488d9f0a65796832ae84 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:39:18.706102 7347 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2eabe017416488d9f0a65796832ae84 [sys.catalog]: This master's current role is: LEARNER
I20260925 04:39:18.709653 7358 catalog_manager.cc:1259] Loaded cluster ID: 9d4b452dab9342578e9f171d3876ca07
I20260925 04:39:18.709713 7358 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d2eabe017416488d9f0a65796832ae84: loading cluster ID for follower catalog manager: success
I20260925 04:39:18.711757 7358 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d2eabe017416488d9f0a65796832ae84: acquiring CA information for follower catalog manager: success
I20260925 04:39:18.713339 7358 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d2eabe017416488d9f0a65796832ae84: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 04:39:18.734449 7270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:39:18.734886 7270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:39:18.734956 7270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 04:39:18.735256 6850 heartbeater.cc:503] Master 127.0.98.62:41387 was elected leader, sending a full tablet report...
W20260925 04:39:18.741454 7270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:39:18.741564 7270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:39:18.741648 7270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 04:39:18.749779 7270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:38919
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35745
--tserver_master_addrs=127.0.98.62:41387,127.0.98.61:46389,127.0.98.60:35759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:39:18.751258 7270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:39:18.752636 7270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:39:18.763522 7363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:18.763578 7364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:18.763554 7366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:18.763629 7270 server_base.cc:1034] running on GCE node
I20260925 04:39:18.764241 7270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:39:18.764961 7270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:39:18.766150 7270 hybrid_clock.cc:648] HybridClock initialized: now 1790311158766126 us; error 36 us; skew 500 ppm
Sep 25 04:39:18 dist-test-slave-dc95 krb5kdc[6480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311158, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:39:18.769119 7270 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260925 04:39:18.770144 7270 webserver.cc:460] Webserver started at http://127.0.98.1:35745/ using document root <none> and password file <none>
I20260925 04:39:18.770717 7270 fs_manager.cc:359] Metadata directory not provided
I20260925 04:39:18.770807 7270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:39:18.775333 7270 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:39:18.778133 7373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:18.779718 7270 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:39:18.779848 7270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-0/wal
uuid: "8a2be5f60c294266a68a6ea6147f2349"
format_stamp: "Formatted at 2026-09-25 04:39:17 on dist-test-slave-dc95"
server_key: "d2c0d86c301d62366e3312f8a3942f4e"
server_key_iv: "193e5e2cd5123d927870eeac044d31b3"
server_key_version: "encryptionkey@0"
I20260925 04:39:18.780454 7270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:39:18.805156 7270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:39:18.808681 7270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:39:18.808857 7270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:39:18.809465 7270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:39:18.810376 7270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:39:18.810426 7270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:18.810467 7270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:39:18.810488 7270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:18.821202 7270 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38919
I20260925 04:39:18.821224 7486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38919 every 8 connection(s)
I20260925 04:39:18.822350 7270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
I20260925 04:39:18.829736 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 7270
I20260925 04:39:18.829944 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 6721
Sep 25 04:39:18 dist-test-slave-dc95 krb5kdc[6480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311158, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 25 04:39:18 dist-test-slave-dc95 krb5kdc[6480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311158, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 25 04:39:18 dist-test-slave-dc95 krb5kdc[6480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311158, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260925 04:39:18.837486 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:42843
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:41387
--webserver_port=41505
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:41387,127.0.98.61:46389
--tserver_master_addrs=127.0.98.62:41387,127.0.98.61:46389,127.0.98.60:35759 with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:39:18.840220 6564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:18.824454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39879 (local address 127.0.98.62:41387)
0925 04:39:18.824567 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:18.824571 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:18.825399 (+ 828us) server_negotiation.cc:407] Connection header received
0925 04:39:18.826691 (+ 1292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:18.826695 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:18.826756 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:18.826840 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:18.829034 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.829826 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.832050 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.832374 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.832453 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:18.836301 (+ 3848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:18.836306 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:18.836309 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:18.836347 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:18.837996 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:18.838539 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:18.838543 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:18.838545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:18.838620 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:18.838921 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:18.838923 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:18.838925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:18.839128 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:18.839214 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:18.839903 (+ 689us) server_negotiation.cc:299] Negotiation successful
0925 04:39:18.840039 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 04:39:18.840220 7248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:18.824269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41163 (local address 127.0.98.61:46389)
0925 04:39:18.824445 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:18.824451 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:18.825402 (+ 951us) server_negotiation.cc:407] Connection header received
0925 04:39:18.826691 (+ 1289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:18.826695 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:18.826768 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:18.826820 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:18.828339 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.828872 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.831105 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.831464 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.831545 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:18.835712 (+ 4167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:18.835717 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:18.835720 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:18.835752 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:18.837934 (+ 2182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:18.838529 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:18.838533 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:18.838535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:18.838620 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:18.838920 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:18.838924 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:18.838925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:18.839128 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:18.839194 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:18.839964 (+ 770us) server_negotiation.cc:299] Negotiation successful
0925 04:39:18.840073 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 04:39:18.840866 7494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:18.825978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35817 (local address 127.0.98.60:35759)
0925 04:39:18.826198 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:18.826202 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:18.826215 (+ 13us) server_negotiation.cc:407] Connection header received
0925 04:39:18.826963 (+ 748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:18.826966 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:18.827020 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:18.827104 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:18.829979 (+ 2875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.830960 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.832846 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.833254 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.833356 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:18.836017 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:18.836021 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:18.836023 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:18.836053 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:18.837512 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:18.838295 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:18.838298 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:18.838300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:18.838389 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:18.838775 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:18.838778 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:18.838780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:18.839032 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:18.839120 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:18.840546 (+ 1426us) server_negotiation.cc:299] Negotiation successful
0925 04:39:18.840699 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":84,"threads_started":1}
I20260925 04:39:18.842504 7493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:18.825832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35759 (local address 127.0.98.1:35817)
0925 04:39:18.826085 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:18.826100 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:18.826413 (+ 313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:18.827133 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:18.827136 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:18.827554 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:18.828186 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:18.828196 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.831519 (+ 3323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.831523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:18.832686 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:18.832699 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.833485 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.833489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:18.833579 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:18.834011 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:18.834024 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:18.835866 (+ 1842us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:39:18.837802 (+ 1936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:18.837808 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:18.837830 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:18.838184 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:18.838481 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:18.838484 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:18.838486 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:18.838667 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:18.839178 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:39:18.839186 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:39:18.839571 (+ 385us) client_negotiation.cc:769] Sending connection context
0925 04:39:18.839823 (+ 252us) client_negotiation.cc:240] Negotiation successful
0925 04:39:18.841074 (+ 1251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":121,"spinlock_wait_cycles":15104,"thread_start_us":90,"threads_started":1}
I20260925 04:39:18.843328 7491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:18.824714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46389 (local address 127.0.98.1:41163)
0925 04:39:18.825218 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:18.825273 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:18.826534 (+ 1261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:18.827014 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:18.827024 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:18.827529 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:18.828177 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:18.828195 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.829027 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.829030 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:18.830973 (+ 1943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:18.830984 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.832023 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.832027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:18.832232 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:18.833137 (+ 905us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:18.833189 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:18.835461 (+ 2272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:39:18.838061 (+ 2600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:18.838064 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:18.838066 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:18.838417 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:18.838733 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:18.838735 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:18.838737 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:18.838832 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:18.839263 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:39:18.839266 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:39:18.839570 (+ 304us) client_negotiation.cc:769] Sending connection context
0925 04:39:18.840421 (+ 851us) client_negotiation.cc:240] Negotiation successful
0925 04:39:18.843140 (+ 2719us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":2546,"spinlock_wait_cycles":44544,"thread_start_us":134,"threads_started":1}
I20260925 04:39:18.843930 7492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:18.824714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41387 (local address 127.0.98.1:39879)
0925 04:39:18.825218 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:18.825274 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:18.826406 (+ 1132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:18.827014 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:18.827022 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:18.827563 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:18.828189 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:18.828196 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.831098 (+ 2902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.831101 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:18.831919 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:18.831929 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:18.832612 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:18.832615 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:18.832714 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:18.833137 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:18.833189 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:18.836127 (+ 2938us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:39:18.838095 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:18.838098 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:18.838100 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:18.838417 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:18.838726 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:18.838729 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:18.838730 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:18.838821 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:18.839263 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:39:18.839266 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:39:18.839570 (+ 304us) client_negotiation.cc:769] Sending connection context
0925 04:39:18.840421 (+ 851us) client_negotiation.cc:240] Negotiation successful
0925 04:39:18.843263 (+ 2842us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"mutex_wait_us":2868,"spinlock_wait_cycles":87424,"thread_start_us":139,"threads_started":1}
I20260925 04:39:18.844064 7487 heartbeater.cc:347] Connected to a master server at 127.0.98.60:35759
I20260925 04:39:18.844456 7487 heartbeater.cc:464] Registering TS with master...
I20260925 04:39:18.844969 7487 heartbeater.cc:511] Master 127.0.98.60:35759 requested a full tablet report, sending...
I20260925 04:39:18.844172 7489 heartbeater.cc:347] Connected to a master server at 127.0.98.61:46389
I20260925 04:39:18.845647 7489 heartbeater.cc:464] Registering TS with master...
I20260925 04:39:18.845795 7488 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41387
I20260925 04:39:18.845834 7489 heartbeater.cc:511] Master 127.0.98.61:46389 requested a full tablet report, sending...
I20260925 04:39:18.845857 7488 heartbeater.cc:464] Registering TS with master...
I20260925 04:39:18.845983 7488 heartbeater.cc:511] Master 127.0.98.62:41387 requested a full tablet report, sending...
I20260925 04:39:18.846707 7308 ts_manager.cc:194] Registered new tserver with Master: 8a2be5f60c294266a68a6ea6147f2349 (127.0.98.1:38919)
I20260925 04:39:18.846786 6524 ts_manager.cc:194] Re-registered known tserver with Master: 8a2be5f60c294266a68a6ea6147f2349 (127.0.98.1:38919)
I20260925 04:39:18.847587 6524 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39879
I20260925 04:39:18.847707 7126 ts_manager.cc:194] Registered new tserver with Master: 8a2be5f60c294266a68a6ea6147f2349 (127.0.98.1:38919)
I20260925 04:39:18.949940 6984 heartbeater.cc:503] Master 127.0.98.62:41387 was elected leader, sending a full tablet report...
W20260925 04:39:18.959213 7495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:39:18.959551 7495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:39:18.959648 7495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:39:18.964107 7495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:39:18.964207 7495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:39:18.964290 7495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 04:39:18.970134 7495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:42843
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=41505
--tserver_master_addrs=127.0.98.62:41387,127.0.98.61:46389,127.0.98.60:35759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7495
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:39:18.971330 7495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:39:18.972529 7495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:39:18.983037 7501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:18.983083 7500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:18.983103 7495 server_base.cc:1034] running on GCE node
W20260925 04:39:18.983364 7503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:18.983944 7495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:39:18.984670 7495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:39:18.985839 7495 hybrid_clock.cc:648] HybridClock initialized: now 1790311158985808 us; error 50 us; skew 500 ppm
Sep 25 04:39:18 dist-test-slave-dc95 krb5kdc[6480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311158, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:39:18.988674 7495 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260925 04:39:18.989718 7495 webserver.cc:460] Webserver started at http://127.0.98.2:41505/ using document root <none> and password file <none>
I20260925 04:39:18.990267 7495 fs_manager.cc:359] Metadata directory not provided
I20260925 04:39:18.990340 7495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:39:18.994547 7495 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:39:18.996755 7510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:18.998174 7495 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:39:18.998315 7495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-1/wal
uuid: "12a3672caea9413f8a824f93100ef42c"
format_stamp: "Formatted at 2026-09-25 04:39:17 on dist-test-slave-dc95"
server_key: "f8d0d0d828c6b9b36a16fce1908ca74a"
server_key_iv: "38d83059c625cca995faafb2eed3a4c5"
server_key_version: "encryptionkey@0"
I20260925 04:39:18.998934 7495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:39:19.014226 7495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:39:19.017661 7495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:39:19.017863 7495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:39:19.018476 7495 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:39:19.019400 7495 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:39:19.019449 7495 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:19.019505 7495 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:39:19.019532 7495 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:19.028641 7495 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42843
I20260925 04:39:19.028661 7623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42843 every 8 connection(s)
I20260925 04:39:19.029896 7495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
I20260925 04:39:19.035770 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 7495
I20260925 04:39:19.035962 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 6855
Sep 25 04:39:19 dist-test-slave-dc95 krb5kdc[6480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311158, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 25 04:39:19 dist-test-slave-dc95 krb5kdc[6480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311158, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 25 04:39:19 dist-test-slave-dc95 krb5kdc[6480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311158, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260925 04:39:19.044327 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:44315
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:41387
--webserver_port=34433
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:41387,127.0.98.61:46389
--tserver_master_addrs=127.0.98.62:41387,127.0.98.61:46389,127.0.98.60:35759 with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:39:19.049337 7494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:19.031926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46021 (local address 127.0.98.60:35759)
0925 04:39:19.032073 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:19.032078 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:19.033557 (+ 1479us) server_negotiation.cc:407] Connection header received
0925 04:39:19.034752 (+ 1195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:19.034755 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:19.034811 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:19.034887 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:19.036675 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:19.037493 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:19.039694 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:19.040060 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:19.040130 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:19.044667 (+ 4537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:19.044671 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:19.044674 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:19.044707 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:19.046609 (+ 1902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:19.047347 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:19.047351 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:19.047352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:19.047436 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:19.047825 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:19.047828 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:19.047830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:19.048098 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:19.048180 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:19.049038 (+ 858us) server_negotiation.cc:299] Negotiation successful
0925 04:39:19.049170 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 04:39:19.049337 6564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:19.032277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50185 (local address 127.0.98.62:41387)
0925 04:39:19.032406 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:19.032411 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:19.033750 (+ 1339us) server_negotiation.cc:407] Connection header received
0925 04:39:19.034867 (+ 1117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:19.034871 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:19.034952 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:19.035025 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:19.036728 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:19.037489 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:19.039698 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:19.040030 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:19.040115 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:19.044714 (+ 4599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:19.044719 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:19.044721 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:19.044751 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:19.047446 (+ 2695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:19.047913 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:19.047917 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:19.047919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:19.048011 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:19.048351 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:19.048354 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:19.048356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:19.048577 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:19.048660 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:19.049039 (+ 379us) server_negotiation.cc:299] Negotiation successful
0925 04:39:19.049167 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 04:39:19.049361 7248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:19.034011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37587 (local address 127.0.98.61:46389)
0925 04:39:19.034131 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:19.034135 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:19.034245 (+ 110us) server_negotiation.cc:407] Connection header received
0925 04:39:19.034751 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:19.034755 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:19.034808 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:19.034855 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:19.036675 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:19.037512 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:19.039694 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:19.040048 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:19.040130 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:19.044497 (+ 4367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:19.044502 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:19.044505 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:19.044535 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:19.046610 (+ 2075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:19.047392 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:19.047397 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:19.047399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:19.047492 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:19.047869 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:19.047872 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:19.047874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:19.048118 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:19.048198 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:19.049030 (+ 832us) server_negotiation.cc:299] Negotiation successful
0925 04:39:19.049166 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 04:39:19.051259 7630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:19.033905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46389 (local address 127.0.98.2:37587)
0925 04:39:19.034138 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:19.034150 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:19.034528 (+ 378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:19.035113 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:19.035123 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:19.035707 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:19.036510 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:19.036516 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:19.037681 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:19.037684 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:19.039511 (+ 1827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:19.039523 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:19.040338 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:19.040342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:19.040644 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:19.041580 (+ 936us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:19.041593 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:19.044180 (+ 2587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:39:19.046760 (+ 2580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:19.046772 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:19.046791 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:19.047244 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:19.047657 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:19.047660 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:19.047663 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:19.047758 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:19.048270 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:39:19.048273 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:39:19.048749 (+ 476us) client_negotiation.cc:769] Sending connection context
0925 04:39:19.049543 (+ 794us) client_negotiation.cc:240] Negotiation successful
0925 04:39:19.050106 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":868,"spinlock_wait_cycles":90880,"thread_start_us":80,"threads_started":1}
I20260925 04:39:19.051256 7628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:19.032514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35759 (local address 127.0.98.2:46021)
0925 04:39:19.033363 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:19.033427 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:19.034528 (+ 1101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:19.035113 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:19.035123 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:19.035706 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:19.036487 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:19.036511 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:19.037681 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:19.037684 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:19.039513 (+ 1829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:19.039522 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:19.040338 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:19.040342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:19.040646 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:19.041479 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:19.041494 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:19.044501 (+ 3007us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:39:19.046761 (+ 2260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:19.046768 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:19.046791 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:19.047220 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:19.047522 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:19.047525 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:19.047527 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:19.047733 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:19.048245 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:39:19.048252 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:39:19.048747 (+ 495us) client_negotiation.cc:769] Sending connection context
0925 04:39:19.050185 (+ 1438us) client_negotiation.cc:240] Negotiation successful
0925 04:39:19.050315 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"mutex_wait_us":457,"spinlock_wait_cycles":52480,"thread_start_us":876,"threads_started":1}
I20260925 04:39:19.051256 7629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:19.032514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41387 (local address 127.0.98.2:50185)
0925 04:39:19.033644 (+ 1130us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:19.033658 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:19.034523 (+ 865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:19.035113 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:19.035123 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:19.035707 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:19.036577 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:19.036585 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:19.037676 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:19.037679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:19.039502 (+ 1823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:19.039516 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:19.040338 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:19.040342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:19.040648 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:19.041459 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:19.041480 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:19.044536 (+ 3056us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:39:19.047566 (+ 3030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:19.047570 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:19.047571 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:19.047817 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:19.048156 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:19.048159 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:19.048161 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:19.048244 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:19.048713 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:39:19.048716 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:39:19.048805 (+ 89us) client_negotiation.cc:769] Sending connection context
0925 04:39:19.049576 (+ 771us) client_negotiation.cc:240] Negotiation successful
0925 04:39:19.050017 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1040,"mutex_wait_us":697,"spinlock_wait_cycles":44160,"thread_start_us":343,"threads_started":1}
I20260925 04:39:19.053855 7625 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41387
I20260925 04:39:19.054219 7625 heartbeater.cc:464] Registering TS with master...
I20260925 04:39:19.055014 7625 heartbeater.cc:511] Master 127.0.98.62:41387 requested a full tablet report, sending...
I20260925 04:39:19.055866 7624 heartbeater.cc:347] Connected to a master server at 127.0.98.60:35759
I20260925 04:39:19.055974 7624 heartbeater.cc:464] Registering TS with master...
I20260925 04:39:19.056048 7626 heartbeater.cc:347] Connected to a master server at 127.0.98.61:46389
I20260925 04:39:19.056146 7626 heartbeater.cc:464] Registering TS with master...
I20260925 04:39:19.056195 7624 heartbeater.cc:511] Master 127.0.98.60:35759 requested a full tablet report, sending...
I20260925 04:39:19.056313 7626 heartbeater.cc:511] Master 127.0.98.61:46389 requested a full tablet report, sending...
I20260925 04:39:19.056700 6524 ts_manager.cc:194] Re-registered known tserver with Master: 12a3672caea9413f8a824f93100ef42c (127.0.98.2:42843)
I20260925 04:39:19.057013 7308 ts_manager.cc:194] Registered new tserver with Master: 12a3672caea9413f8a824f93100ef42c (127.0.98.2:42843)
I20260925 04:39:19.057013 7126 ts_manager.cc:194] Registered new tserver with Master: 12a3672caea9413f8a824f93100ef42c (127.0.98.2:42843)
I20260925 04:39:19.057526 6524 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50185
I20260925 04:39:19.118278 7082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:19.113644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35759 (local address 127.0.0.1:55170)
0925 04:39:19.113961 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:19.113985 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:19.114334 (+ 349us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:19.114752 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:19.114756 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:19.114786 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:39:19.114895 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:19.114901 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:19.116029 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:19.116033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:19.116967 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:19.116984 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:19.117922 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:19.117925 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:19.118001 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:19.118026 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 04:39:19.118077 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 04:39:19.118154 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158}
I20260925 04:39:19.118312 7494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:19.113908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55170 (local address 127.0.98.60:35759)
0925 04:39:19.114119 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:19.114125 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:19.114151 (+ 26us) server_negotiation.cc:407] Connection header received
0925 04:39:19.114504 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:19.114508 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:19.114581 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:19.114696 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:39:19.115010 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:19.115821 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:19.117122 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:19.117765 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:19.117892 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:19.118062 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:19.118107 (+ 45us) server_negotiation.cc:299] Negotiation successful
0925 04:39:19.118179 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 04:39:19.119184 7318 raft_consensus.cc:1203] T 00000000000000000000000000000000 P d2eabe017416488d9f0a65796832ae84 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260925 04:39:19.119889 7347 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2eabe017416488d9f0a65796832ae84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36a07e678247497c977b44bb1cf41c03. Latest consensus state: current_term: 1 leader_uuid: "36a07e678247497c977b44bb1cf41c03" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "36a07e678247497c977b44bb1cf41c03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41387 } } peers { permanent_uuid: "50cabbb535294fa09b1dac97495cde70" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46389 } attrs { promote: false } } peers { permanent_uuid: "d2eabe017416488d9f0a65796832ae84" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35759 } attrs { promote: true } } }
I20260925 04:39:19.120060 7347 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2eabe017416488d9f0a65796832ae84 [sys.catalog]: This master's current role is: LEARNER
W20260925 04:39:19.171841 7631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:39:19.172153 7631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:39:19.172219 7631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:39:19.176885 7631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:39:19.177007 7631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:39:19.177098 7631 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 04:39:19.183148 7631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:44315
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=34433
--tserver_master_addrs=127.0.98.62:41387,127.0.98.61:46389,127.0.98.60:35759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:39:19.184482 7631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:39:19.185616 7631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:39:19.196367 7637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:19.196331 7639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:19.196381 7636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:19.196354 7631 server_base.cc:1034] running on GCE node
I20260925 04:39:19.196993 7631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:39:19.197682 7631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:39:19.198856 7631 hybrid_clock.cc:648] HybridClock initialized: now 1790311159198829 us; error 40 us; skew 500 ppm
Sep 25 04:39:19 dist-test-slave-dc95 krb5kdc[6480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311159, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:39:19.201597 7631 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260925 04:39:19.202582 7631 webserver.cc:460] Webserver started at http://127.0.98.3:34433/ using document root <none> and password file <none>
I20260925 04:39:19.203151 7631 fs_manager.cc:359] Metadata directory not provided
I20260925 04:39:19.203231 7631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:39:19.207763 7631 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:39:19.210054 7646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:19.211390 7631 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:39:19.211530 7631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-2/wal
uuid: "6682fed6b70747e791e1cf424b939759"
format_stamp: "Formatted at 2026-09-25 04:39:17 on dist-test-slave-dc95"
server_key: "b5da8a4da97b1b8cb9a1de1e3c2ae4bf"
server_key_iv: "dc5f9d20f68b52a60965ae0678253f97"
server_key_version: "encryptionkey@0"
I20260925 04:39:19.212181 7631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:39:19.228825 7631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:39:19.232002 7631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:39:19.232187 7631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:39:19.232797 7631 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:39:19.233664 7631 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:39:19.233711 7631 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:19.233762 7631 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:39:19.233791 7631 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:19.242604 7631 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44315
I20260925 04:39:19.242622 7759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44315 every 8 connection(s)
I20260925 04:39:19.243732 7631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
I20260925 04:39:19.253175 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 7631
Sep 25 04:39:19 dist-test-slave-dc95 krb5kdc[6480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311159, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 25 04:39:19 dist-test-slave-dc95 krb5kdc[6480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311159, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 25 04:39:19 dist-test-slave-dc95 krb5kdc[6480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311159, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 04:39:19.264042 7248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:19.247475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40659 (local address 127.0.98.61:46389)
0925 04:39:19.247612 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:19.247615 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:19.247629 (+ 14us) server_negotiation.cc:407] Connection header received
0925 04:39:19.248103 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:19.248107 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:19.248163 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:19.248234 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:19.249911 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:19.250694 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:19.252032 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:19.252353 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:19.252434 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:19.258304 (+ 5870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:19.258308 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:19.258311 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:19.258339 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:19.261290 (+ 2951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:19.261987 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:19.261991 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:19.261994 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:19.262076 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:19.262524 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:19.262527 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:19.262529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:19.262759 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:19.262848 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:19.263733 (+ 885us) server_negotiation.cc:299] Negotiation successful
0925 04:39:19.263861 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 04:39:19.265199 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:19.246155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41387 (local address 127.0.98.3:49041)
0925 04:39:19.247003 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:19.247016 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:19.247888 (+ 872us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:19.248552 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:19.248559 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:19.249043 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:19.250000 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:19.250010 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:19.251018 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:19.251021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:19.251825 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:19.251834 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:19.252518 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:19.252522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:19.252633 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:19.253694 (+ 1061us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:19.253708 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:19.258574 (+ 4866us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:39:19.262302 (+ 3728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:19.262305 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:19.262307 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:19.262550 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:19.262890 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:19.262893 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:19.262895 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:19.262983 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:19.263447 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:39:19.263450 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:39:19.263545 (+ 95us) client_negotiation.cc:769] Sending connection context
0925 04:39:19.263657 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 04:39:19.264037 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":763,"mutex_wait_us":102,"spinlock_wait_cycles":45056,"thread_start_us":141,"threads_started":1}
I20260925 04:39:19.265352 7771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:19.255267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36218 (local address 127.0.98.61:46389)
0925 04:39:19.255807 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:19.255815 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:19.255962 (+ 147us) server_negotiation.cc:407] Connection header received
0925 04:39:19.256231 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:19.256234 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:19.256288 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:19.256370 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:19.257438 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:19.258199 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:19.259426 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:19.259806 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:19.259902 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:19.261306 (+ 1404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:19.261310 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:19.261313 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:19.261358 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:19.262950 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:19.263375 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:19.263378 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:19.263380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:19.263567 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:19.263935 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:19.263939 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:19.263940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:19.264130 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:19.264403 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:19.265079 (+ 676us) server_negotiation.cc:299] Negotiation successful
0925 04:39:19.265199 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":94,"threads_started":1}
I20260925 04:39:19.265479 7766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:19.247134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46389 (local address 127.0.98.3:40659)
0925 04:39:19.247389 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:19.247402 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:19.247891 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:19.248553 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:19.248561 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:19.249065 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:19.249722 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:19.249743 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:19.250813 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:19.250816 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:19.251913 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:19.251919 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:19.252548 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:19.252551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:19.252672 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:19.253469 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:19.253484 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:19.258115 (+ 4631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:39:19.261415 (+ 3300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:19.261430 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:19.261452 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:19.261868 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:19.262179 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:19.262182 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:19.262184 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:19.262400 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:19.262904 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:39:19.262912 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:39:19.263282 (+ 370us) client_negotiation.cc:769] Sending connection context
0925 04:39:19.263573 (+ 291us) client_negotiation.cc:240] Negotiation successful
0925 04:39:19.264561 (+ 988us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":163,"spinlock_wait_cycles":3072,"thread_start_us":74,"threads_started":1}
I20260925 04:39:19.266031 7775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:19.255833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55178 (local address 127.0.98.60:35759)
0925 04:39:19.256143 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:19.256148 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:19.256540 (+ 392us) server_negotiation.cc:407] Connection header received
0925 04:39:19.256702 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:19.256706 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:19.256764 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:19.256828 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:19.257704 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:19.258479 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:19.259702 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:19.260058 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:19.260137 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:19.262325 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:19.262329 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:19.262332 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:19.262357 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:19.263966 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:19.264408 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:19.264412 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:19.264414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:19.264493 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:19.264782 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:19.264785 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:19.264786 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:19.265013 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:19.265106 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:19.265754 (+ 648us) server_negotiation.cc:299] Negotiation successful
0925 04:39:19.265874 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":159,"threads_started":1}
I20260925 04:39:19.266412 6564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:19.245721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49041 (local address 127.0.98.62:41387)
0925 04:39:19.245888 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:19.245893 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:19.247105 (+ 1212us) server_negotiation.cc:407] Connection header received
0925 04:39:19.248241 (+ 1136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:19.248245 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:19.248298 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:19.248367 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:19.250149 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:19.250924 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:19.251943 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:19.252276 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:19.252375 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:19.258736 (+ 6361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:19.258741 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:19.258743 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:19.258771 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:19.262186 (+ 3415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:19.262666 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:19.262669 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:19.262671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:19.262783 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:19.263084 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:19.263088 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:19.263090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:19.263313 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:19.263402 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:19.266133 (+ 2731us) server_negotiation.cc:299] Negotiation successful
0925 04:39:19.266269 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 04:39:19.266564 7762 heartbeater.cc:347] Connected to a master server at 127.0.98.61:46389
I20260925 04:39:19.266881 7762 heartbeater.cc:464] Registering TS with master...
I20260925 04:39:19.266955 7761 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41387
I20260925 04:39:19.267035 7761 heartbeater.cc:464] Registering TS with master...
I20260925 04:39:19.267516 7761 heartbeater.cc:511] Master 127.0.98.62:41387 requested a full tablet report, sending...
I20260925 04:39:19.268143 7762 heartbeater.cc:511] Master 127.0.98.61:46389 requested a full tablet report, sending...
I20260925 04:39:19.268787 6524 ts_manager.cc:194] Re-registered known tserver with Master: 6682fed6b70747e791e1cf424b939759 (127.0.98.3:44315)
I20260925 04:39:19.268927 7126 ts_manager.cc:194] Registered new tserver with Master: 6682fed6b70747e791e1cf424b939759 (127.0.98.3:44315)
I20260925 04:39:19.269547 6524 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49041
I20260925 04:39:19.269728 7494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:19.245762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45839 (local address 127.0.98.60:35759)
0925 04:39:19.245888 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:19.245893 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:19.246914 (+ 1021us) server_negotiation.cc:407] Connection header received
0925 04:39:19.248162 (+ 1248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:19.248165 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:19.248235 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:19.248282 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:19.249899 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:19.250438 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:19.252054 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:19.252289 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:19.252380 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:19.259479 (+ 7099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:19.259482 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:19.259485 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:19.259514 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:19.261214 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:19.268404 (+ 7190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:19.268408 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:19.268410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:19.268503 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:19.268806 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:19.268809 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:19.268811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:19.269020 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:19.269099 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:19.269454 (+ 355us) server_negotiation.cc:299] Negotiation successful
0925 04:39:19.269582 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 04:39:19.270332 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:19.246155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35759 (local address 127.0.98.3:45839)
0925 04:39:19.246725 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:19.246797 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:19.247895 (+ 1098us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:19.248553 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:19.248560 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:19.249044 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:19.249722 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:19.249743 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:19.250582 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:19.250585 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:19.251908 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:19.251919 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:19.252444 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:19.252448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:19.252632 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:19.253426 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:19.253456 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:19.259318 (+ 5862us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:39:19.267974 (+ 8656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:19.267977 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:19.267979 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:19.268289 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:19.268603 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:19.268605 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:19.268607 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:19.268694 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:19.269154 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:39:19.269157 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:39:19.269268 (+ 111us) client_negotiation.cc:769] Sending connection context
0925 04:39:19.269976 (+ 708us) client_negotiation.cc:240] Negotiation successful
0925 04:39:19.270158 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"mutex_wait_us":100,"spinlock_wait_cycles":30720,"thread_start_us":607,"threads_started":1}
I20260925 04:39:19.271071 7760 heartbeater.cc:347] Connected to a master server at 127.0.98.60:35759
I20260925 04:39:19.271168 7760 heartbeater.cc:464] Registering TS with master...
I20260925 04:39:19.271324 7760 heartbeater.cc:511] Master 127.0.98.60:35759 requested a full tablet report, sending...
I20260925 04:39:19.272033 7308 ts_manager.cc:194] Registered new tserver with Master: 6682fed6b70747e791e1cf424b939759 (127.0.98.3:44315)
I20260925 04:39:19.275184 7773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:19.255438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49582 (local address 127.0.98.62:41387)
0925 04:39:19.256659 (+ 1221us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:19.256665 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:19.257331 (+ 666us) server_negotiation.cc:407] Connection header received
0925 04:39:19.257384 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:19.257387 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:19.257440 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:19.257514 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:19.267038 (+ 9524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:19.267840 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:19.269031 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:19.269354 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:19.269452 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:19.271808 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:19.271812 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:19.271815 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:19.271844 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:19.273319 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:19.273771 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:19.273774 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:19.273775 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:19.273896 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:19.274306 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:19.274309 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:19.274310 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:19.274558 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:19.274668 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:19.274892 (+ 224us) server_negotiation.cc:299] Negotiation successful
0925 04:39:19.275019 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2195,"server-negotiator.queue_time_us":1083,"thread_start_us":141,"threads_started":1}
I20260925 04:39:19.278376 6524 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49582:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:39:19.281580 6524 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:39:19.284314 7084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36a07e678247497c977b44bb1cf41c03 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "36a07e678247497c977b44bb1cf41c03" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "36a07e678247497c977b44bb1cf41c03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41387 } } peers { permanent_uuid: "50cabbb535294fa09b1dac97495cde70" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46389 } attrs { promote: false } } peers { permanent_uuid: "d2eabe017416488d9f0a65796832ae84" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35759 } attrs { promote: true } } }
I20260925 04:39:19.284528 7084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36a07e678247497c977b44bb1cf41c03 [sys.catalog]: This master's current role is: LEADER
I20260925 04:39:19.287689 7084 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 36a07e678247497c977b44bb1cf41c03: attempting to promote NON_VOTER d2eabe017416488d9f0a65796832ae84 to VOTER
I20260925 04:39:19.288002 7084 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36a07e678247497c977b44bb1cf41c03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "36a07e678247497c977b44bb1cf41c03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41387 } } peers { permanent_uuid: "50cabbb535294fa09b1dac97495cde70" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46389 } attrs { promote: false } } peers { permanent_uuid: "d2eabe017416488d9f0a65796832ae84" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35759 } attrs { promote: false } }
I20260925 04:39:19.290397 7136 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 50cabbb535294fa09b1dac97495cde70 [term 1 FOLLOWER]: Refusing update from remote peer 36a07e678247497c977b44bb1cf41c03: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260925 04:39:19.290711 7318 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d2eabe017416488d9f0a65796832ae84 [term 1 LEARNER]: Refusing update from remote peer 36a07e678247497c977b44bb1cf41c03: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260925 04:39:19.290814 7780 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 36a07e678247497c977b44bb1cf41c03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50cabbb535294fa09b1dac97495cde70" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46389 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260925 04:39:19.291340 7780 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 36a07e678247497c977b44bb1cf41c03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2eabe017416488d9f0a65796832ae84" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260925 04:39:19.292474 7778 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 36a07e678247497c977b44bb1cf41c03 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, d2eabe017416488d9f0a65796832ae84 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "36a07e678247497c977b44bb1cf41c03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41387 } } peers { permanent_uuid: "50cabbb535294fa09b1dac97495cde70" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46389 } attrs { promote: false } } peers { permanent_uuid: "d2eabe017416488d9f0a65796832ae84" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35759 } attrs { promote: false } } }
I20260925 04:39:19.292976 7136 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 50cabbb535294fa09b1dac97495cde70 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, d2eabe017416488d9f0a65796832ae84 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "36a07e678247497c977b44bb1cf41c03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41387 } } peers { permanent_uuid: "50cabbb535294fa09b1dac97495cde70" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46389 } attrs { promote: false } } peers { permanent_uuid: "d2eabe017416488d9f0a65796832ae84" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35759 } attrs { promote: false } } }
I20260925 04:39:19.294235 7778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36a07e678247497c977b44bb1cf41c03 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "36a07e678247497c977b44bb1cf41c03" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "36a07e678247497c977b44bb1cf41c03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41387 } } peers { permanent_uuid: "50cabbb535294fa09b1dac97495cde70" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46389 } attrs { promote: false } } peers { permanent_uuid: "d2eabe017416488d9f0a65796832ae84" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35759 } attrs { promote: false } } }
I20260925 04:39:19.294351 7778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36a07e678247497c977b44bb1cf41c03 [sys.catalog]: This master's current role is: LEADER
I20260925 04:39:19.294518 7778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36a07e678247497c977b44bb1cf41c03 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "36a07e678247497c977b44bb1cf41c03" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "36a07e678247497c977b44bb1cf41c03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41387 } } peers { permanent_uuid: "50cabbb535294fa09b1dac97495cde70" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46389 } attrs { promote: false } } peers { permanent_uuid: "d2eabe017416488d9f0a65796832ae84" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35759 } attrs { promote: false } } }
I20260925 04:39:19.294597 7778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36a07e678247497c977b44bb1cf41c03 [sys.catalog]: This master's current role is: LEADER
I20260925 04:39:19.294801 7778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36a07e678247497c977b44bb1cf41c03 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "36a07e678247497c977b44bb1cf41c03" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "36a07e678247497c977b44bb1cf41c03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41387 } } peers { permanent_uuid: "50cabbb535294fa09b1dac97495cde70" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46389 } attrs { promote: false } } peers { permanent_uuid: "d2eabe017416488d9f0a65796832ae84" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35759 } attrs { promote: false } } }
I20260925 04:39:19.294888 7778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36a07e678247497c977b44bb1cf41c03 [sys.catalog]: This master's current role is: LEADER
I20260925 04:39:19.294585 7318 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d2eabe017416488d9f0a65796832ae84 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, d2eabe017416488d9f0a65796832ae84 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "36a07e678247497c977b44bb1cf41c03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41387 } } peers { permanent_uuid: "50cabbb535294fa09b1dac97495cde70" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46389 } attrs { promote: false } } peers { permanent_uuid: "d2eabe017416488d9f0a65796832ae84" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35759 } attrs { promote: false } } }
I20260925 04:39:19.297040 7347 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2eabe017416488d9f0a65796832ae84 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "36a07e678247497c977b44bb1cf41c03" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "36a07e678247497c977b44bb1cf41c03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41387 } } peers { permanent_uuid: "50cabbb535294fa09b1dac97495cde70" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46389 } attrs { promote: false } } peers { permanent_uuid: "d2eabe017416488d9f0a65796832ae84" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35759 } attrs { promote: false } } }
I20260925 04:39:19.297169 7347 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2eabe017416488d9f0a65796832ae84 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:39:19.299975 7788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50cabbb535294fa09b1dac97495cde70 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "36a07e678247497c977b44bb1cf41c03" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "36a07e678247497c977b44bb1cf41c03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41387 } } peers { permanent_uuid: "50cabbb535294fa09b1dac97495cde70" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46389 } attrs { promote: false } } peers { permanent_uuid: "d2eabe017416488d9f0a65796832ae84" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35759 } attrs { promote: false } } }
I20260925 04:39:19.300148 7788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50cabbb535294fa09b1dac97495cde70 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:39:19.302507 7082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:19.296717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38919 (local address 127.0.0.1:47224)
0925 04:39:19.296860 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:19.296877 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:19.296975 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:19.298505 (+ 1530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:19.298509 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:19.298529 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:39:19.298624 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:19.298629 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:19.299835 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:19.299839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:19.300886 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:19.300910 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:19.302058 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:19.302061 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:19.302155 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:19.302180 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 04:39:19.302275 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 04:39:19.302363 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 04:39:19.302570 7790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:19.297583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47224 (local address 127.0.98.1:38919)
0925 04:39:19.297916 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:19.297932 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:19.297957 (+ 25us) server_negotiation.cc:407] Connection header received
0925 04:39:19.298048 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:19.298054 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:19.298316 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:19.298756 (+ 440us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:39:19.298870 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:19.299742 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:19.301029 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:19.301902 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:19.302019 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:19.302189 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:19.302311 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 04:39:19.302421 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":86,"threads_started":1}
I20260925 04:39:19.303913 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:19.298283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47462 (local address 127.0.98.2:42843)
0925 04:39:19.299049 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:19.299058 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:19.299077 (+ 19us) server_negotiation.cc:407] Connection header received
0925 04:39:19.299136 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:19.299140 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:19.299298 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:19.299415 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:39:19.299681 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:19.300527 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:19.302045 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:19.303071 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:19.303180 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:19.303293 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:19.303632 (+ 339us) server_negotiation.cc:299] Negotiation successful
0925 04:39:19.303735 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":684,"thread_start_us":96,"threads_started":1}
I20260925 04:39:19.305992 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:19.298069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44110 (local address 127.0.98.3:44315)
0925 04:39:19.300457 (+ 2388us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:19.300468 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:19.300493 (+ 25us) server_negotiation.cc:407] Connection header received
0925 04:39:19.300569 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:19.300574 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:19.300778 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:19.302108 (+ 1330us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:39:19.302264 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:19.303173 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:19.304678 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:19.305391 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:19.305495 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:19.305643 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:19.305742 (+ 99us) server_negotiation.cc:299] Negotiation successful
0925 04:39:19.305860 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2261,"thread_start_us":111,"threads_started":1}
I20260925 04:39:19.306394 7421 tablet_service.cc:1467] Processing CreateTablet for tablet d312d8345b164ab69b84414ecac5ce53 (DEFAULT_TABLE table=test-table [id=02a3b17f28fb496ba87c938ad27f7bf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:39:19.307543 7421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d312d8345b164ab69b84414ecac5ce53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:39:19.303813 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:19.298113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42843 (local address 127.0.0.1:47462)
0925 04:39:19.298807 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:19.298816 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:19.298862 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:19.299437 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:19.299440 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:19.299460 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:39:19.299548 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:19.299553 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:19.300642 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:19.300645 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:19.301897 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:19.301906 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:19.303351 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:19.303355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:19.303430 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:19.303483 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 04:39:19.303548 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 04:39:19.303665 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"mutex_wait_us":118,"thread_start_us":89,"threads_started":1}
I20260925 04:39:19.307870 7789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:19.297265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44315 (local address 127.0.0.1:44110)
0925 04:39:19.298598 (+ 1333us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:19.298610 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:19.298683 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:19.300934 (+ 2251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:19.300937 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:19.300956 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:39:19.301062 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:19.301067 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:19.303306 (+ 2239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:19.303310 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:19.304469 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:19.304482 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:19.305498 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:19.305502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:19.305591 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:19.305613 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 04:39:19.305676 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 04:39:19.305755 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1273,"mutex_wait_us":1,"thread_start_us":115,"threads_started":1}
I20260925 04:39:19.311182 7694 tablet_service.cc:1467] Processing CreateTablet for tablet d312d8345b164ab69b84414ecac5ce53 (DEFAULT_TABLE table=test-table [id=02a3b17f28fb496ba87c938ad27f7bf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:39:19.312471 7694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d312d8345b164ab69b84414ecac5ce53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:39:19.313582 7558 tablet_service.cc:1467] Processing CreateTablet for tablet d312d8345b164ab69b84414ecac5ce53 (DEFAULT_TABLE table=test-table [id=02a3b17f28fb496ba87c938ad27f7bf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:39:19.314586 7558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d312d8345b164ab69b84414ecac5ce53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:39:19.316826 7794 tablet_bootstrap.cc:492] T d312d8345b164ab69b84414ecac5ce53 P 8a2be5f60c294266a68a6ea6147f2349: Bootstrap starting.
I20260925 04:39:19.319382 7794 tablet_bootstrap.cc:654] T d312d8345b164ab69b84414ecac5ce53 P 8a2be5f60c294266a68a6ea6147f2349: Neither blocks nor log segments found. Creating new log.
I20260925 04:39:19.320374 7794 log.cc:824] T d312d8345b164ab69b84414ecac5ce53 P 8a2be5f60c294266a68a6ea6147f2349: Log is configured to *not* fsync() on all Append() calls
I20260925 04:39:19.320883 7795 tablet_bootstrap.cc:492] T d312d8345b164ab69b84414ecac5ce53 P 6682fed6b70747e791e1cf424b939759: Bootstrap starting.
I20260925 04:39:19.322606 7794 tablet_bootstrap.cc:492] T d312d8345b164ab69b84414ecac5ce53 P 8a2be5f60c294266a68a6ea6147f2349: No bootstrap required, opened a new log
I20260925 04:39:19.322824 7794 ts_tablet_manager.cc:1397] T d312d8345b164ab69b84414ecac5ce53 P 8a2be5f60c294266a68a6ea6147f2349: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260925 04:39:19.323728 7795 tablet_bootstrap.cc:654] T d312d8345b164ab69b84414ecac5ce53 P 6682fed6b70747e791e1cf424b939759: Neither blocks nor log segments found. Creating new log.
I20260925 04:39:19.324048 7797 tablet_bootstrap.cc:492] T d312d8345b164ab69b84414ecac5ce53 P 12a3672caea9413f8a824f93100ef42c: Bootstrap starting.
I20260925 04:39:19.324522 7795 log.cc:824] T d312d8345b164ab69b84414ecac5ce53 P 6682fed6b70747e791e1cf424b939759: Log is configured to *not* fsync() on all Append() calls
I20260925 04:39:19.326231 7795 tablet_bootstrap.cc:492] T d312d8345b164ab69b84414ecac5ce53 P 6682fed6b70747e791e1cf424b939759: No bootstrap required, opened a new log
I20260925 04:39:19.326272 7797 tablet_bootstrap.cc:654] T d312d8345b164ab69b84414ecac5ce53 P 12a3672caea9413f8a824f93100ef42c: Neither blocks nor log segments found. Creating new log.
I20260925 04:39:19.326413 7795 ts_tablet_manager.cc:1397] T d312d8345b164ab69b84414ecac5ce53 P 6682fed6b70747e791e1cf424b939759: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 04:39:19.326705 7794 raft_consensus.cc:348] T d312d8345b164ab69b84414ecac5ce53 P 8a2be5f60c294266a68a6ea6147f2349 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a2be5f60c294266a68a6ea6147f2349" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38919 } } peers { permanent_uuid: "6682fed6b70747e791e1cf424b939759" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44315 } } peers { permanent_uuid: "12a3672caea9413f8a824f93100ef42c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42843 } }
I20260925 04:39:19.327116 7794 raft_consensus.cc:374] T d312d8345b164ab69b84414ecac5ce53 P 8a2be5f60c294266a68a6ea6147f2349 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:39:19.327181 7794 raft_consensus.cc:729] T d312d8345b164ab69b84414ecac5ce53 P 8a2be5f60c294266a68a6ea6147f2349 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a2be5f60c294266a68a6ea6147f2349, State: Initialized, Role: FOLLOWER
I20260925 04:39:19.327247 7797 log.cc:824] T d312d8345b164ab69b84414ecac5ce53 P 12a3672caea9413f8a824f93100ef42c: Log is configured to *not* fsync() on all Append() calls
I20260925 04:39:19.327751 7794 consensus_queue.cc:260] T d312d8345b164ab69b84414ecac5ce53 P 8a2be5f60c294266a68a6ea6147f2349 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a2be5f60c294266a68a6ea6147f2349" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38919 } } peers { permanent_uuid: "6682fed6b70747e791e1cf424b939759" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44315 } } peers { permanent_uuid: "12a3672caea9413f8a824f93100ef42c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42843 } }
I20260925 04:39:19.329007 7794 ts_tablet_manager.cc:1428] T d312d8345b164ab69b84414ecac5ce53 P 8a2be5f60c294266a68a6ea6147f2349: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 04:39:19.329196 7488 heartbeater.cc:503] Master 127.0.98.62:41387 was elected leader, sending a full tablet report...
I20260925 04:39:19.329659 7797 tablet_bootstrap.cc:492] T d312d8345b164ab69b84414ecac5ce53 P 12a3672caea9413f8a824f93100ef42c: No bootstrap required, opened a new log
I20260925 04:39:19.329941 7797 ts_tablet_manager.cc:1397] T d312d8345b164ab69b84414ecac5ce53 P 12a3672caea9413f8a824f93100ef42c: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260925 04:39:19.331514 7795 raft_consensus.cc:348] T d312d8345b164ab69b84414ecac5ce53 P 6682fed6b70747e791e1cf424b939759 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a2be5f60c294266a68a6ea6147f2349" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38919 } } peers { permanent_uuid: "6682fed6b70747e791e1cf424b939759" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44315 } } peers { permanent_uuid: "12a3672caea9413f8a824f93100ef42c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42843 } }
I20260925 04:39:19.331868 7795 raft_consensus.cc:374] T d312d8345b164ab69b84414ecac5ce53 P 6682fed6b70747e791e1cf424b939759 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:39:19.331936 7795 raft_consensus.cc:729] T d312d8345b164ab69b84414ecac5ce53 P 6682fed6b70747e791e1cf424b939759 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6682fed6b70747e791e1cf424b939759, State: Initialized, Role: FOLLOWER
I20260925 04:39:19.332659 7795 consensus_queue.cc:260] T d312d8345b164ab69b84414ecac5ce53 P 6682fed6b70747e791e1cf424b939759 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a2be5f60c294266a68a6ea6147f2349" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38919 } } peers { permanent_uuid: "6682fed6b70747e791e1cf424b939759" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44315 } } peers { permanent_uuid: "12a3672caea9413f8a824f93100ef42c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42843 } }
I20260925 04:39:19.333830 7795 ts_tablet_manager.cc:1428] T d312d8345b164ab69b84414ecac5ce53 P 6682fed6b70747e791e1cf424b939759: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260925 04:39:19.335155 7761 heartbeater.cc:503] Master 127.0.98.62:41387 was elected leader, sending a full tablet report...
I20260925 04:39:19.336930 7797 raft_consensus.cc:348] T d312d8345b164ab69b84414ecac5ce53 P 12a3672caea9413f8a824f93100ef42c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a2be5f60c294266a68a6ea6147f2349" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38919 } } peers { permanent_uuid: "6682fed6b70747e791e1cf424b939759" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44315 } } peers { permanent_uuid: "12a3672caea9413f8a824f93100ef42c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42843 } }
I20260925 04:39:19.337257 7797 raft_consensus.cc:374] T d312d8345b164ab69b84414ecac5ce53 P 12a3672caea9413f8a824f93100ef42c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:39:19.337317 7797 raft_consensus.cc:729] T d312d8345b164ab69b84414ecac5ce53 P 12a3672caea9413f8a824f93100ef42c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12a3672caea9413f8a824f93100ef42c, State: Initialized, Role: FOLLOWER
I20260925 04:39:19.337944 7797 consensus_queue.cc:260] T d312d8345b164ab69b84414ecac5ce53 P 12a3672caea9413f8a824f93100ef42c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a2be5f60c294266a68a6ea6147f2349" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38919 } } peers { permanent_uuid: "6682fed6b70747e791e1cf424b939759" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44315 } } peers { permanent_uuid: "12a3672caea9413f8a824f93100ef42c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42843 } }
I20260925 04:39:19.339089 7797 ts_tablet_manager.cc:1428] T d312d8345b164ab69b84414ecac5ce53 P 12a3672caea9413f8a824f93100ef42c: Time spent starting tablet: real 0.009s user 0.000s sys 0.006s
I20260925 04:39:19.339299 7625 heartbeater.cc:503] Master 127.0.98.62:41387 was elected leader, sending a full tablet report...
W20260925 04:39:19.495252 7763 tablet.cc:2369] T d312d8345b164ab69b84414ecac5ce53 P 6682fed6b70747e791e1cf424b939759: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:39:19.528654 7802 raft_consensus.cc:482] T d312d8345b164ab69b84414ecac5ce53 P 12a3672caea9413f8a824f93100ef42c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:39:19.528847 7802 raft_consensus.cc:504] T d312d8345b164ab69b84414ecac5ce53 P 12a3672caea9413f8a824f93100ef42c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a2be5f60c294266a68a6ea6147f2349" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38919 } } peers { permanent_uuid: "6682fed6b70747e791e1cf424b939759" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44315 } } peers { permanent_uuid: "12a3672caea9413f8a824f93100ef42c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42843 } }
I20260925 04:39:19.530051 7802 leader_election.cc:288] T d312d8345b164ab69b84414ecac5ce53 P 12a3672caea9413f8a824f93100ef42c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a2be5f60c294266a68a6ea6147f2349 (127.0.98.1:38919), 6682fed6b70747e791e1cf424b939759 (127.0.98.3:44315)
I20260925 04:39:19.533720 7790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:19.530393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60443 (local address 127.0.98.1:38919)
0925 04:39:19.530502 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:19.530507 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:19.530521 (+ 14us) server_negotiation.cc:407] Connection header received
0925 04:39:19.530656 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:19.530660 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:19.530713 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:19.530784 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:39:19.531037 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:19.531530 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:19.532737 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:19.533156 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:19.533268 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:19.533338 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:19.533530 (+ 192us) server_negotiation.cc:299] Negotiation successful
0925 04:39:19.533614 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260925 04:39:19.533731 7627 tablet.cc:2369] T d312d8345b164ab69b84414ecac5ce53 P 12a3672caea9413f8a824f93100ef42c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:39:19.533888 7629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:19.530316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44315 (local address 127.0.98.2:57705)
0925 04:39:19.530753 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:19.530765 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:19.530885 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:19.531105 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:19.531108 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:19.531122 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:39:19.531218 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:19.531223 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:19.531988 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:19.531991 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:19.532786 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:19.532795 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:19.533483 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:19.533487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:19.533586 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:19.533607 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 04:39:19.533675 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 04:39:19.533744 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"mutex_wait_us":328}
I20260925 04:39:19.534016 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:19.530434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57705 (local address 127.0.98.3:44315)
0925 04:39:19.530554 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:19.530557 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:19.530871 (+ 314us) server_negotiation.cc:407] Connection header received
0925 04:39:19.530985 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:19.530987 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:19.531028 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:19.531088 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:39:19.531362 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:19.531890 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:19.532899 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:19.533357 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:19.533460 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:19.533520 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:19.533788 (+ 268us) server_negotiation.cc:299] Negotiation successful
0925 04:39:19.533868 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 04:39:19.533720 7630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:19.530308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38919 (local address 127.0.98.2:60443)
0925 04:39:19.530437 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:19.530454 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:19.530547 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:19.530799 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:19.530801 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:19.530819 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:39:19.530936 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:19.530941 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:19.531661 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:19.531665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:19.532608 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:19.532618 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:19.533286 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:19.533290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:19.533392 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:19.533414 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 04:39:19.533489 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 04:39:19.533574 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":23}
I20260925 04:39:19.534655 7714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d312d8345b164ab69b84414ecac5ce53" candidate_uuid: "12a3672caea9413f8a824f93100ef42c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6682fed6b70747e791e1cf424b939759" is_pre_election: true
I20260925 04:39:19.534845 7441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d312d8345b164ab69b84414ecac5ce53" candidate_uuid: "12a3672caea9413f8a824f93100ef42c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a2be5f60c294266a68a6ea6147f2349" is_pre_election: true
I20260925 04:39:19.534962 7714 raft_consensus.cc:2454] T d312d8345b164ab69b84414ecac5ce53 P 6682fed6b70747e791e1cf424b939759 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12a3672caea9413f8a824f93100ef42c in term 0.
I20260925 04:39:19.535121 7441 raft_consensus.cc:2454] T d312d8345b164ab69b84414ecac5ce53 P 8a2be5f60c294266a68a6ea6147f2349 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12a3672caea9413f8a824f93100ef42c in term 0.
I20260925 04:39:19.535482 7514 leader_election.cc:302] T d312d8345b164ab69b84414ecac5ce53 P 12a3672caea9413f8a824f93100ef42c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 12a3672caea9413f8a824f93100ef42c, 6682fed6b70747e791e1cf424b939759, 8a2be5f60c294266a68a6ea6147f2349; no voters:
I20260925 04:39:19.535815 7802 raft_consensus.cc:2781] T d312d8345b164ab69b84414ecac5ce53 P 12a3672caea9413f8a824f93100ef42c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:39:19.535898 7802 raft_consensus.cc:482] T d312d8345b164ab69b84414ecac5ce53 P 12a3672caea9413f8a824f93100ef42c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:39:19.535974 7802 raft_consensus.cc:3037] T d312d8345b164ab69b84414ecac5ce53 P 12a3672caea9413f8a824f93100ef42c [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:39:19.536995 7802 raft_consensus.cc:504] T d312d8345b164ab69b84414ecac5ce53 P 12a3672caea9413f8a824f93100ef42c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a2be5f60c294266a68a6ea6147f2349" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38919 } } peers { permanent_uuid: "6682fed6b70747e791e1cf424b939759" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44315 } } peers { permanent_uuid: "12a3672caea9413f8a824f93100ef42c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42843 } }
I20260925 04:39:19.537353 7802 leader_election.cc:288] T d312d8345b164ab69b84414ecac5ce53 P 12a3672caea9413f8a824f93100ef42c [CANDIDATE]: Term 1 election: Requested vote from peers 8a2be5f60c294266a68a6ea6147f2349 (127.0.98.1:38919), 6682fed6b70747e791e1cf424b939759 (127.0.98.3:44315)
I20260925 04:39:19.537766 7714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d312d8345b164ab69b84414ecac5ce53" candidate_uuid: "12a3672caea9413f8a824f93100ef42c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6682fed6b70747e791e1cf424b939759"
I20260925 04:39:19.537919 7714 raft_consensus.cc:3037] T d312d8345b164ab69b84414ecac5ce53 P 6682fed6b70747e791e1cf424b939759 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:39:19.538954 7714 raft_consensus.cc:2454] T d312d8345b164ab69b84414ecac5ce53 P 6682fed6b70747e791e1cf424b939759 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12a3672caea9413f8a824f93100ef42c in term 1.
I20260925 04:39:19.539228 7441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d312d8345b164ab69b84414ecac5ce53" candidate_uuid: "12a3672caea9413f8a824f93100ef42c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a2be5f60c294266a68a6ea6147f2349"
I20260925 04:39:19.539268 7514 leader_election.cc:302] T d312d8345b164ab69b84414ecac5ce53 P 12a3672caea9413f8a824f93100ef42c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12a3672caea9413f8a824f93100ef42c, 6682fed6b70747e791e1cf424b939759; no voters:
I20260925 04:39:19.539374 7441 raft_consensus.cc:3037] T d312d8345b164ab69b84414ecac5ce53 P 8a2be5f60c294266a68a6ea6147f2349 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:39:19.539497 7802 raft_consensus.cc:2781] T d312d8345b164ab69b84414ecac5ce53 P 12a3672caea9413f8a824f93100ef42c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:39:19.539701 7802 raft_consensus.cc:686] T d312d8345b164ab69b84414ecac5ce53 P 12a3672caea9413f8a824f93100ef42c [term 1 LEADER]: Becoming Leader. State: Replica: 12a3672caea9413f8a824f93100ef42c, State: Running, Role: LEADER
I20260925 04:39:19.540346 7802 consensus_queue.cc:237] T d312d8345b164ab69b84414ecac5ce53 P 12a3672caea9413f8a824f93100ef42c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a2be5f60c294266a68a6ea6147f2349" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38919 } } peers { permanent_uuid: "6682fed6b70747e791e1cf424b939759" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44315 } } peers { permanent_uuid: "12a3672caea9413f8a824f93100ef42c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42843 } }
I20260925 04:39:19.540483 7441 raft_consensus.cc:2454] T d312d8345b164ab69b84414ecac5ce53 P 8a2be5f60c294266a68a6ea6147f2349 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12a3672caea9413f8a824f93100ef42c in term 1.
I20260925 04:39:19.543787 6524 catalog_manager.cc:5495] T d312d8345b164ab69b84414ecac5ce53 P 12a3672caea9413f8a824f93100ef42c reported cstate change: term changed from 0 to 1, leader changed from <none> to 12a3672caea9413f8a824f93100ef42c (127.0.98.2). New cstate: current_term: 1 leader_uuid: "12a3672caea9413f8a824f93100ef42c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a2be5f60c294266a68a6ea6147f2349" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6682fed6b70747e791e1cf424b939759" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12a3672caea9413f8a824f93100ef42c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42843 } health_report { overall_health: HEALTHY } } }
W20260925 04:39:19.574160 7490 tablet.cc:2369] T d312d8345b164ab69b84414ecac5ce53 P 8a2be5f60c294266a68a6ea6147f2349: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:39:19.593291 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:19.589694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47464 (local address 127.0.98.2:42843)
0925 04:39:19.589821 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:19.589825 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:19.589839 (+ 14us) server_negotiation.cc:407] Connection header received
0925 04:39:19.590009 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:19.590012 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:19.590071 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:19.590162 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:39:19.590405 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:19.590924 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:19.591799 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:19.592043 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:19.592135 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:19.592423 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:39:19.592859 (+ 436us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:39:19.592980 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:19.593059 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 04:39:19.593124 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 04:39:19.599191 7441 raft_consensus.cc:1261] T d312d8345b164ab69b84414ecac5ce53 P 8a2be5f60c294266a68a6ea6147f2349 [term 1 FOLLOWER]: Refusing update from remote peer 12a3672caea9413f8a824f93100ef42c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:39:19.599191 7714 raft_consensus.cc:1261] T d312d8345b164ab69b84414ecac5ce53 P 6682fed6b70747e791e1cf424b939759 [term 1 FOLLOWER]: Refusing update from remote peer 12a3672caea9413f8a824f93100ef42c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:39:19.599918 7803 consensus_queue.cc:1035] T d312d8345b164ab69b84414ecac5ce53 P 12a3672caea9413f8a824f93100ef42c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a2be5f60c294266a68a6ea6147f2349" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:39:19.600106 7802 consensus_queue.cc:1035] T d312d8345b164ab69b84414ecac5ce53 P 12a3672caea9413f8a824f93100ef42c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6682fed6b70747e791e1cf424b939759" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:39:19.606153 7806 mvcc.cc:205] Tried to move back new op lower bound from 7333114509711134720 to 7333114509480161280. Current Snapshot: MvccSnapshot[applied={T|T < 7333114509711134720}]
I20260925 04:39:19.610282 7808 mvcc.cc:205] Tried to move back new op lower bound from 7333114509711134720 to 7333114509480161280. Current Snapshot: MvccSnapshot[applied={T|T < 7333114509711134720}]
I20260925 04:39:19.615008 7807 mvcc.cc:205] Tried to move back new op lower bound from 7333114509711134720 to 7333114509480161280. Current Snapshot: MvccSnapshot[applied={T|T < 7333114509711134720}]
I20260925 04:39:19.616766 6524 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49582:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:39:19.616966 6524 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49582:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:39:19.622066 6524 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 36a07e678247497c977b44bb1cf41c03: Sending DeleteTablet for 3 replicas of tablet d312d8345b164ab69b84414ecac5ce53
I20260925 04:39:19.622761 7694 tablet_service.cc:1514] Processing DeleteTablet for tablet d312d8345b164ab69b84414ecac5ce53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:39:19 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44110
I20260925 04:39:19.622972 7558 tablet_service.cc:1514] Processing DeleteTablet for tablet d312d8345b164ab69b84414ecac5ce53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:39:19 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47462
I20260925 04:39:19.622978 7557 tablet_service.cc:1514] Processing DeleteTablet for tablet d312d8345b164ab69b84414ecac5ce53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:39:19 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47462
I20260925 04:39:19.623768 7421 tablet_service.cc:1514] Processing DeleteTablet for tablet d312d8345b164ab69b84414ecac5ce53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:39:19 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47224
I20260925 04:39:19.623548 7816 tablet_replica.cc:330] T d312d8345b164ab69b84414ecac5ce53 P 6682fed6b70747e791e1cf424b939759: stopping tablet replica
I20260925 04:39:19.624478 7816 raft_consensus.cc:2229] T d312d8345b164ab69b84414ecac5ce53 P 6682fed6b70747e791e1cf424b939759 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:39:19.624496 7817 tablet_replica.cc:330] T d312d8345b164ab69b84414ecac5ce53 P 12a3672caea9413f8a824f93100ef42c: stopping tablet replica
I20260925 04:39:19.624776 7817 raft_consensus.cc:2229] T d312d8345b164ab69b84414ecac5ce53 P 12a3672caea9413f8a824f93100ef42c [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:39:19.624886 7816 raft_consensus.cc:2258] T d312d8345b164ab69b84414ecac5ce53 P 6682fed6b70747e791e1cf424b939759 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:39:19.624984 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 6493
I20260925 04:39:19.625124 7817 raft_consensus.cc:2258] T d312d8345b164ab69b84414ecac5ce53 P 12a3672caea9413f8a824f93100ef42c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:39:19.625921 7819 tablet_replica.cc:330] T d312d8345b164ab69b84414ecac5ce53 P 8a2be5f60c294266a68a6ea6147f2349: stopping tablet replica
I20260925 04:39:19.626195 7819 raft_consensus.cc:2229] T d312d8345b164ab69b84414ecac5ce53 P 8a2be5f60c294266a68a6ea6147f2349 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:39:19.626459 7819 raft_consensus.cc:2258] T d312d8345b164ab69b84414ecac5ce53 P 8a2be5f60c294266a68a6ea6147f2349 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:39:19.627692 7819 ts_tablet_manager.cc:1905] T d312d8345b164ab69b84414ecac5ce53 P 8a2be5f60c294266a68a6ea6147f2349: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:39:19.627687 7816 ts_tablet_manager.cc:1905] T d312d8345b164ab69b84414ecac5ce53 P 6682fed6b70747e791e1cf424b939759: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:39:19.630412 7819 ts_tablet_manager.cc:1918] T d312d8345b164ab69b84414ecac5ce53 P 8a2be5f60c294266a68a6ea6147f2349: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:39:19.630412 7816 ts_tablet_manager.cc:1918] T d312d8345b164ab69b84414ecac5ce53 P 6682fed6b70747e791e1cf424b939759: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:39:19.630529 7816 log.cc:1196] T d312d8345b164ab69b84414ecac5ce53 P 6682fed6b70747e791e1cf424b939759: Deleting WAL directory at /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-2/wal/wals/d312d8345b164ab69b84414ecac5ce53
I20260925 04:39:19.630671 7819 log.cc:1196] T d312d8345b164ab69b84414ecac5ce53 P 8a2be5f60c294266a68a6ea6147f2349: Deleting WAL directory at /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-0/wal/wals/d312d8345b164ab69b84414ecac5ce53
I20260925 04:39:19.630837 7816 ts_tablet_manager.cc:1939] T d312d8345b164ab69b84414ecac5ce53 P 6682fed6b70747e791e1cf424b939759: Deleting consensus metadata
I20260925 04:39:19.631258 7819 ts_tablet_manager.cc:1939] T d312d8345b164ab69b84414ecac5ce53 P 8a2be5f60c294266a68a6ea6147f2349: Deleting consensus metadata
W20260925 04:39:19.632419 7374 connection.cc:441] server connection from 127.0.0.1:47224 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47224 (request call id 1) could send its response
W20260925 04:39:19.632405 7650 connection.cc:441] server connection from 127.0.0.1:44110 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44110 (request call id 1) could send its response
I20260925 04:39:19.636329 7817 ts_tablet_manager.cc:1905] T d312d8345b164ab69b84414ecac5ce53 P 12a3672caea9413f8a824f93100ef42c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:39:19.639809 7817 ts_tablet_manager.cc:1918] T d312d8345b164ab69b84414ecac5ce53 P 12a3672caea9413f8a824f93100ef42c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:39:19.639930 7817 log.cc:1196] T d312d8345b164ab69b84414ecac5ce53 P 12a3672caea9413f8a824f93100ef42c: Deleting WAL directory at /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-1/wal/wals/d312d8345b164ab69b84414ecac5ce53
I20260925 04:39:19.640259 7817 ts_tablet_manager.cc:1939] T d312d8345b164ab69b84414ecac5ce53 P 12a3672caea9413f8a824f93100ef42c: Deleting consensus metadata
W20260925 04:39:19.641533 7511 connection.cc:441] server connection from 127.0.0.1:47462 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47462 (request call id 1) could send its response
W20260925 04:39:19.642453 7511 connection.cc:441] server connection from 127.0.0.1:47462 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47462 (request call id 2) could send its response
I20260925 04:39:19.644197 7248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:19.633701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36244 (local address 127.0.98.61:46389)
0925 04:39:19.633829 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:19.633833 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:19.634138 (+ 305us) server_negotiation.cc:407] Connection header received
0925 04:39:19.634319 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:19.634322 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:19.634363 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:19.634407 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:19.635661 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:19.636385 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:19.637622 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:19.637937 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:19.638022 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:19.639867 (+ 1845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:19.639871 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:19.639873 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:19.639894 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:19.642146 (+ 2252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:19.642799 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:19.642803 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:19.642805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:19.642905 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:19.643224 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:19.643227 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:19.643228 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:19.643417 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:19.643508 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:19.643901 (+ 393us) server_negotiation.cc:299] Negotiation successful
0925 04:39:19.644030 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 04:39:19.645967 7775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:19.634716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55194 (local address 127.0.98.60:35759)
0925 04:39:19.634839 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:19.634843 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:19.634971 (+ 128us) server_negotiation.cc:407] Connection header received
0925 04:39:19.635087 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:19.635090 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:19.635142 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:19.635216 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:19.636099 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:19.636813 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:19.638746 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:19.639077 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:19.639162 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:19.642257 (+ 3095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:19.642261 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:19.642264 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:19.642294 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:19.644235 (+ 1941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:19.644702 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:19.644706 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:19.644707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:19.644793 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:19.645074 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:19.645077 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:19.645078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:19.645279 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:19.645374 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:19.645717 (+ 343us) server_negotiation.cc:299] Negotiation successful
0925 04:39:19.645846 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 04:39:19.646531 7771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:19.623656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36228 (local address 127.0.98.61:46389)
0925 04:39:19.624155 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:19.624159 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:19.632802 (+ 8643us) server_negotiation.cc:407] Connection header received
0925 04:39:19.632915 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:19.632918 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:19.632968 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:19.633051 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:19.633829 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:19.634568 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:19.639304 (+ 4736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:19.639697 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:19.639772 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:19.642780 (+ 3008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:19.642784 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:19.642787 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:19.642815 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:19.644799 (+ 1984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:19.645269 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:19.645272 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:19.645274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:19.645353 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:19.645642 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:19.645645 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:19.645647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:19.645842 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:19.645932 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:19.646240 (+ 308us) server_negotiation.cc:299] Negotiation successful
0925 04:39:19.646375 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"spinlock_wait_cycles":7040}
I20260925 04:39:19.655105 7494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:19.623959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55182 (local address 127.0.98.60:35759)
0925 04:39:19.624065 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:19.624069 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:19.647085 (+ 23016us) server_negotiation.cc:407] Connection header received
0925 04:39:19.647292 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:19.647295 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:19.647341 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:19.647419 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:19.648203 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:19.648803 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:19.649919 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:19.650262 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:19.650331 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:19.652103 (+ 1772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:19.652106 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:19.652108 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:19.652131 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:19.653435 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:19.653870 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:19.653872 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:19.653874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:19.653946 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:19.654226 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:19.654228 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:19.654229 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:19.654418 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:19.654537 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:19.654820 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 04:39:19.654939 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 04:39:20.615758 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:20.615230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41387 (local address 127.0.98.3:55783)
0925 04:39:20.615526 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:20.615653 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":87,"threads_started":1}
W20260925 04:39:20.616057 7761 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41387 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:41387: connect: Connection refused (error 111)
I20260925 04:39:20.624322 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:20.623894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41387 (local address 127.0.98.1:38633)
0925 04:39:20.624140 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:20.624225 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":66,"threads_started":1}
W20260925 04:39:20.624689 7488 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41387 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:41387: connect: Connection refused (error 111)
I20260925 04:39:20.624758 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:20.624295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41387 (local address 127.0.98.2:43967)
0925 04:39:20.624580 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:20.624663 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":63,"threads_started":1}
W20260925 04:39:20.625007 7625 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41387 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:41387: connect: Connection refused (error 111)
I20260925 04:39:21.121776 7830 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50cabbb535294fa09b1dac97495cde70 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 36a07e678247497c977b44bb1cf41c03)
I20260925 04:39:21.121999 7830 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50cabbb535294fa09b1dac97495cde70 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "36a07e678247497c977b44bb1cf41c03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41387 } } peers { permanent_uuid: "50cabbb535294fa09b1dac97495cde70" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46389 } attrs { promote: false } } peers { permanent_uuid: "d2eabe017416488d9f0a65796832ae84" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35759 } attrs { promote: false } }
I20260925 04:39:21.122376 7831 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2eabe017416488d9f0a65796832ae84 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 36a07e678247497c977b44bb1cf41c03)
I20260925 04:39:21.122534 7831 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2eabe017416488d9f0a65796832ae84 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "36a07e678247497c977b44bb1cf41c03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41387 } } peers { permanent_uuid: "50cabbb535294fa09b1dac97495cde70" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46389 } attrs { promote: false } } peers { permanent_uuid: "d2eabe017416488d9f0a65796832ae84" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35759 } attrs { promote: false } }
I20260925 04:39:21.123286 7830 leader_election.cc:288] T 00000000000000000000000000000000 P 50cabbb535294fa09b1dac97495cde70 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 36a07e678247497c977b44bb1cf41c03 (127.0.98.62:41387), d2eabe017416488d9f0a65796832ae84 (127.0.98.60:35759)
I20260925 04:39:21.123402 7831 leader_election.cc:288] T 00000000000000000000000000000000 P d2eabe017416488d9f0a65796832ae84 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 36a07e678247497c977b44bb1cf41c03 (127.0.98.62:41387), 50cabbb535294fa09b1dac97495cde70 (127.0.98.61:46389)
I20260925 04:39:21.123937 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:21.123446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41387 (local address 127.0.0.1:50068)
0925 04:39:21.123751 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:21.123833 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":58,"threads_started":1}
W20260925 04:39:21.124119 7111 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41387: connect: Connection refused (error 111)
I20260925 04:39:21.124519 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:21.123884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41387 (local address 127.0.0.1:50082)
0925 04:39:21.124363 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:21.124434 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":44,"threads_started":1}
W20260925 04:39:21.124688 7293 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41387: connect: Connection refused (error 111)
I20260925 04:39:21.125131 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:21.124890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41387 (local address 127.0.0.1:50092)
0925 04:39:21.125011 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:21.125056 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260925 04:39:21.125224 7111 leader_election.cc:334] T 00000000000000000000000000000000 P 50cabbb535294fa09b1dac97495cde70 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 36a07e678247497c977b44bb1cf41c03 (127.0.98.62:41387): Network error: Client connection negotiation failed: client connection to 127.0.98.62:41387: connect: Connection refused (error 111)
I20260925 04:39:21.125676 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:21.125460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41387 (local address 127.0.0.1:50096)
0925 04:39:21.125566 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:21.125616 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 04:39:21.125849 7293 leader_election.cc:334] T 00000000000000000000000000000000 P d2eabe017416488d9f0a65796832ae84 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 36a07e678247497c977b44bb1cf41c03 (127.0.98.62:41387): Network error: Client connection negotiation failed: client connection to 127.0.98.62:41387: connect: Connection refused (error 111)
I20260925 04:39:21.127426 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:21.123755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36254 (local address 127.0.98.61:46389)
0925 04:39:21.123950 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:21.123954 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:21.123968 (+ 14us) server_negotiation.cc:407] Connection header received
0925 04:39:21.124090 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:21.124094 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:21.124140 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:21.124212 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:39:21.124582 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:21.125319 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:21.126536 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:21.126931 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:21.127021 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:21.127088 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:21.127248 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 04:39:21.127317 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":45,"threads_started":1}
I20260925 04:39:21.127426 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:21.123585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46389 (local address 127.0.0.1:36254)
0925 04:39:21.123850 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:21.123865 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:21.123971 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:21.124325 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:21.124328 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:21.124350 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:39:21.124450 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:21.124457 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:21.125437 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:21.125441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:21.126422 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:21.126431 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:21.127042 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:21.127045 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:21.127133 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:21.127161 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 04:39:21.127226 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 04:39:21.127302 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":54,"threads_started":1}
I20260925 04:39:21.127872 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:21.124032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55204 (local address 127.0.98.60:35759)
0925 04:39:21.124416 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:21.124420 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:21.124432 (+ 12us) server_negotiation.cc:407] Connection header received
0925 04:39:21.124470 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:21.124473 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:21.124510 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:21.125014 (+ 504us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:39:21.125105 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:21.125882 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:21.126893 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:21.127290 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:21.127375 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:21.127426 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:21.127705 (+ 279us) server_negotiation.cc:299] Negotiation successful
0925 04:39:21.127774 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":331,"thread_start_us":58,"threads_started":1}
I20260925 04:39:21.127871 7136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d2eabe017416488d9f0a65796832ae84" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "50cabbb535294fa09b1dac97495cde70" is_pre_election: true
I20260925 04:39:21.128064 7136 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 50cabbb535294fa09b1dac97495cde70 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2eabe017416488d9f0a65796832ae84 in term 1.
I20260925 04:39:21.128296 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:21.123520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35759 (local address 127.0.0.1:55204)
0925 04:39:21.124071 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:21.124084 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:21.124187 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:21.124765 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:21.124768 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:21.124787 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:39:21.124882 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:21.124889 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:21.126002 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:21.126006 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:21.126778 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:21.126786 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:21.127393 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:21.127397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:21.127484 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:21.127507 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 04:39:21.127578 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 04:39:21.127665 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"mutex_wait_us":1,"thread_start_us":103,"threads_started":1}
I20260925 04:39:21.128403 7294 leader_election.cc:302] T 00000000000000000000000000000000 P d2eabe017416488d9f0a65796832ae84 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 50cabbb535294fa09b1dac97495cde70, d2eabe017416488d9f0a65796832ae84; no voters: 36a07e678247497c977b44bb1cf41c03
I20260925 04:39:21.128630 7831 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2eabe017416488d9f0a65796832ae84 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 04:39:21.128645 7318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "50cabbb535294fa09b1dac97495cde70" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d2eabe017416488d9f0a65796832ae84" is_pre_election: true
I20260925 04:39:21.128708 7831 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2eabe017416488d9f0a65796832ae84 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 36a07e678247497c977b44bb1cf41c03)
I20260925 04:39:21.128757 7831 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2eabe017416488d9f0a65796832ae84 [term 1 FOLLOWER]: Advancing to term 2
I20260925 04:39:21.129748 7831 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2eabe017416488d9f0a65796832ae84 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "36a07e678247497c977b44bb1cf41c03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41387 } } peers { permanent_uuid: "50cabbb535294fa09b1dac97495cde70" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46389 } attrs { promote: false } } peers { permanent_uuid: "d2eabe017416488d9f0a65796832ae84" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35759 } attrs { promote: false } }
I20260925 04:39:21.130007 7318 raft_consensus.cc:2379] T 00000000000000000000000000000000 P d2eabe017416488d9f0a65796832ae84 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 50cabbb535294fa09b1dac97495cde70 in current term 2: Already voted for candidate d2eabe017416488d9f0a65796832ae84 in this term.
I20260925 04:39:21.130119 7831 leader_election.cc:288] T 00000000000000000000000000000000 P d2eabe017416488d9f0a65796832ae84 [CANDIDATE]: Term 2 election: Requested vote from peers 36a07e678247497c977b44bb1cf41c03 (127.0.98.62:41387), 50cabbb535294fa09b1dac97495cde70 (127.0.98.61:46389)
I20260925 04:39:21.130492 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:21.130250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41387 (local address 127.0.0.1:50108)
0925 04:39:21.130388 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:21.130441 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260925 04:39:21.130481 7136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d2eabe017416488d9f0a65796832ae84" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "50cabbb535294fa09b1dac97495cde70"
I20260925 04:39:21.130656 7136 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50cabbb535294fa09b1dac97495cde70 [term 1 FOLLOWER]: Advancing to term 2
I20260925 04:39:21.131590 7136 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 50cabbb535294fa09b1dac97495cde70 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2eabe017416488d9f0a65796832ae84 in term 2.
I20260925 04:39:21.131781 7112 leader_election.cc:302] T 00000000000000000000000000000000 P 50cabbb535294fa09b1dac97495cde70 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 50cabbb535294fa09b1dac97495cde70; no voters: 36a07e678247497c977b44bb1cf41c03, d2eabe017416488d9f0a65796832ae84
I20260925 04:39:21.131882 7294 leader_election.cc:302] T 00000000000000000000000000000000 P d2eabe017416488d9f0a65796832ae84 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50cabbb535294fa09b1dac97495cde70, d2eabe017416488d9f0a65796832ae84; no voters:
I20260925 04:39:21.132018 7831 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2eabe017416488d9f0a65796832ae84 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 04:39:21.132054 7830 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 50cabbb535294fa09b1dac97495cde70 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 04:39:21.132131 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:21.131863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41387 (local address 127.0.0.1:50114)
0925 04:39:21.132013 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:21.132068 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260925 04:39:21.132241 7831 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2eabe017416488d9f0a65796832ae84 [term 2 LEADER]: Becoming Leader. State: Replica: d2eabe017416488d9f0a65796832ae84, State: Running, Role: LEADER
W20260925 04:39:21.132225 7293 leader_election.cc:334] T 00000000000000000000000000000000 P d2eabe017416488d9f0a65796832ae84 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 36a07e678247497c977b44bb1cf41c03 (127.0.98.62:41387): Network error: Client connection negotiation failed: client connection to 127.0.98.62:41387: connect: Connection refused (error 111)
I20260925 04:39:21.132516 7831 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2eabe017416488d9f0a65796832ae84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "36a07e678247497c977b44bb1cf41c03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41387 } } peers { permanent_uuid: "50cabbb535294fa09b1dac97495cde70" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46389 } attrs { promote: false } } peers { permanent_uuid: "d2eabe017416488d9f0a65796832ae84" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35759 } attrs { promote: false } }
I20260925 04:39:21.133715 7840 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2eabe017416488d9f0a65796832ae84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2eabe017416488d9f0a65796832ae84. Latest consensus state: current_term: 2 leader_uuid: "d2eabe017416488d9f0a65796832ae84" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "36a07e678247497c977b44bb1cf41c03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41387 } } peers { permanent_uuid: "50cabbb535294fa09b1dac97495cde70" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46389 } attrs { promote: false } } peers { permanent_uuid: "d2eabe017416488d9f0a65796832ae84" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35759 } attrs { promote: false } } }
I20260925 04:39:21.133842 7840 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2eabe017416488d9f0a65796832ae84 [sys.catalog]: This master's current role is: LEADER
I20260925 04:39:21.134114 7842 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:39:21.136438 7842 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:39:21.136868 7842 catalog_manager.cc:1259] Loaded cluster ID: 9d4b452dab9342578e9f171d3876ca07
I20260925 04:39:21.136916 7842 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:39:21.137476 7842 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:39:21.137866 7842 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d2eabe017416488d9f0a65796832ae84: Loaded TSK: 0
I20260925 04:39:21.138581 7842 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:39:21.159245 7307 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55194:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:39:21.162441 7307 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:39:21.165699 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:21.165369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41387 (local address 127.0.0.1:50134)
0925 04:39:21.165478 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:21.165600 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260925 04:39:21.165733 7136 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 50cabbb535294fa09b1dac97495cde70 [term 2 FOLLOWER]: Refusing update from remote peer d2eabe017416488d9f0a65796832ae84: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260925 04:39:21.166344 7840 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d2eabe017416488d9f0a65796832ae84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50cabbb535294fa09b1dac97495cde70" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46389 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260925 04:39:21.166478 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:21.166249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41387 (local address 127.0.0.1:50150)
0925 04:39:21.166353 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:21.166411 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 04:39:21.166585 7293 consensus_peers.cc:487] T 00000000000000000000000000000000 P d2eabe017416488d9f0a65796832ae84 -> Peer 36a07e678247497c977b44bb1cf41c03 (127.0.98.62:41387): Couldn't send request to peer 36a07e678247497c977b44bb1cf41c03. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41387: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 04:39:21.168196 7830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50cabbb535294fa09b1dac97495cde70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2eabe017416488d9f0a65796832ae84. Latest consensus state: current_term: 2 leader_uuid: "d2eabe017416488d9f0a65796832ae84" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "36a07e678247497c977b44bb1cf41c03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41387 } } peers { permanent_uuid: "50cabbb535294fa09b1dac97495cde70" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46389 } attrs { promote: false } } peers { permanent_uuid: "d2eabe017416488d9f0a65796832ae84" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35759 } attrs { promote: false } } }
I20260925 04:39:21.168356 7830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50cabbb535294fa09b1dac97495cde70 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:39:21.169535 7831 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2eabe017416488d9f0a65796832ae84 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "d2eabe017416488d9f0a65796832ae84" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "36a07e678247497c977b44bb1cf41c03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41387 } } peers { permanent_uuid: "50cabbb535294fa09b1dac97495cde70" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46389 } attrs { promote: false } } peers { permanent_uuid: "d2eabe017416488d9f0a65796832ae84" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35759 } attrs { promote: false } } }
I20260925 04:39:21.169734 7831 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2eabe017416488d9f0a65796832ae84 [sys.catalog]: This master's current role is: LEADER
I20260925 04:39:21.170773 7830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50cabbb535294fa09b1dac97495cde70 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "d2eabe017416488d9f0a65796832ae84" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "36a07e678247497c977b44bb1cf41c03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41387 } } peers { permanent_uuid: "50cabbb535294fa09b1dac97495cde70" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46389 } attrs { promote: false } } peers { permanent_uuid: "d2eabe017416488d9f0a65796832ae84" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35759 } attrs { promote: false } } }
I20260925 04:39:21.170913 7830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50cabbb535294fa09b1dac97495cde70 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:39:21.180450 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:21.175920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44315 (local address 127.0.0.1:44116)
0925 04:39:21.176437 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:21.176451 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:21.176617 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:21.176966 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:21.176969 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:21.176990 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:39:21.177091 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:21.177097 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:21.178215 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:21.178219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:21.179086 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:21.179095 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:21.179907 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:21.179911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:21.179995 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:21.180142 (+ 147us) client_negotiation.cc:769] Sending connection context
0925 04:39:21.180217 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 04:39:21.180296 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"mutex_wait_us":100}
I20260925 04:39:21.180864 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:21.176507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38919 (local address 127.0.0.1:47228)
0925 04:39:21.176688 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:21.176700 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:21.176773 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:21.177316 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:21.177319 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:21.177335 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:39:21.177414 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:21.177420 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:21.178440 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:21.178443 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:21.179677 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:21.179685 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:21.180521 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:21.180525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:21.180614 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:21.180638 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 04:39:21.180699 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 04:39:21.180769 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120}
I20260925 04:39:21.180891 7849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:21.176050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44116 (local address 127.0.98.3:44315)
0925 04:39:21.176406 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:21.176416 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:21.176570 (+ 154us) server_negotiation.cc:407] Connection header received
0925 04:39:21.176726 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:21.176729 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:21.176779 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:21.176898 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:39:21.177240 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:21.178047 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:21.179265 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:21.179788 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:21.179895 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:21.179967 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:21.180660 (+ 693us) server_negotiation.cc:299] Negotiation successful
0925 04:39:21.180753 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":67,"threads_started":1}
I20260925 04:39:21.181500 7850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:21.176787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47228 (local address 127.0.98.1:38919)
0925 04:39:21.177083 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:21.177088 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:21.177102 (+ 14us) server_negotiation.cc:407] Connection header received
0925 04:39:21.177150 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:21.177152 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:21.177199 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:21.177515 (+ 316us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:39:21.177620 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:21.178354 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:21.179791 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:21.180384 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:21.181197 (+ 813us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:21.181265 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:21.181296 (+ 31us) server_negotiation.cc:299] Negotiation successful
0925 04:39:21.181368 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":100,"threads_started":1}
I20260925 04:39:21.181447 7694 tablet_service.cc:1467] Processing CreateTablet for tablet 86e243d770314d2d85cbbfc6e8264925 (DEFAULT_TABLE table=test-table [id=f31962847017453bb1b95f62b52154c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:39:21.181768 7694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86e243d770314d2d85cbbfc6e8264925. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:39:21.182461 7851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:21.177262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42843 (local address 127.0.0.1:47470)
0925 04:39:21.177877 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:21.177888 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:21.177956 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:21.178308 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:21.178311 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:21.178326 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:39:21.178424 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:21.178430 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:21.179571 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:21.179574 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:21.181266 (+ 1692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:21.181275 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:21.182050 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:21.182054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:21.182140 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:21.182166 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 04:39:21.182252 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 04:39:21.182329 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"mutex_wait_us":24,"thread_start_us":107,"threads_started":1}
I20260925 04:39:21.182596 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:21.177629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47470 (local address 127.0.98.2:42843)
0925 04:39:21.178110 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:21.178113 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:21.178124 (+ 11us) server_negotiation.cc:407] Connection header received
0925 04:39:21.178165 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:21.178167 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:21.178203 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:21.178275 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:39:21.178544 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:21.179444 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:21.181406 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:21.181925 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:21.182017 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:21.182068 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:21.182366 (+ 298us) server_negotiation.cc:299] Negotiation successful
0925 04:39:21.182472 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":66,"threads_started":1}
I20260925 04:39:21.183419 7557 tablet_service.cc:1467] Processing CreateTablet for tablet 86e243d770314d2d85cbbfc6e8264925 (DEFAULT_TABLE table=test-table [id=f31962847017453bb1b95f62b52154c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:39:21.183751 7557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86e243d770314d2d85cbbfc6e8264925. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:39:21.184043 7421 tablet_service.cc:1467] Processing CreateTablet for tablet 86e243d770314d2d85cbbfc6e8264925 (DEFAULT_TABLE table=test-table [id=f31962847017453bb1b95f62b52154c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:39:21.184329 7421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86e243d770314d2d85cbbfc6e8264925. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:39:21.184593 7853 tablet_bootstrap.cc:492] T 86e243d770314d2d85cbbfc6e8264925 P 6682fed6b70747e791e1cf424b939759: Bootstrap starting.
I20260925 04:39:21.185565 7853 tablet_bootstrap.cc:654] T 86e243d770314d2d85cbbfc6e8264925 P 6682fed6b70747e791e1cf424b939759: Neither blocks nor log segments found. Creating new log.
I20260925 04:39:21.186218 7854 tablet_bootstrap.cc:492] T 86e243d770314d2d85cbbfc6e8264925 P 12a3672caea9413f8a824f93100ef42c: Bootstrap starting.
I20260925 04:39:21.186892 7856 tablet_bootstrap.cc:492] T 86e243d770314d2d85cbbfc6e8264925 P 8a2be5f60c294266a68a6ea6147f2349: Bootstrap starting.
I20260925 04:39:21.187074 7853 tablet_bootstrap.cc:492] T 86e243d770314d2d85cbbfc6e8264925 P 6682fed6b70747e791e1cf424b939759: No bootstrap required, opened a new log
I20260925 04:39:21.187139 7854 tablet_bootstrap.cc:654] T 86e243d770314d2d85cbbfc6e8264925 P 12a3672caea9413f8a824f93100ef42c: Neither blocks nor log segments found. Creating new log.
I20260925 04:39:21.187155 7853 ts_tablet_manager.cc:1397] T 86e243d770314d2d85cbbfc6e8264925 P 6682fed6b70747e791e1cf424b939759: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 04:39:21.187829 7856 tablet_bootstrap.cc:654] T 86e243d770314d2d85cbbfc6e8264925 P 8a2be5f60c294266a68a6ea6147f2349: Neither blocks nor log segments found. Creating new log.
I20260925 04:39:21.188016 7853 raft_consensus.cc:348] T 86e243d770314d2d85cbbfc6e8264925 P 6682fed6b70747e791e1cf424b939759 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6682fed6b70747e791e1cf424b939759" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44315 } } peers { permanent_uuid: "12a3672caea9413f8a824f93100ef42c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42843 } } peers { permanent_uuid: "8a2be5f60c294266a68a6ea6147f2349" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38919 } }
I20260925 04:39:21.188139 7853 raft_consensus.cc:374] T 86e243d770314d2d85cbbfc6e8264925 P 6682fed6b70747e791e1cf424b939759 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:39:21.188171 7853 raft_consensus.cc:729] T 86e243d770314d2d85cbbfc6e8264925 P 6682fed6b70747e791e1cf424b939759 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6682fed6b70747e791e1cf424b939759, State: Initialized, Role: FOLLOWER
I20260925 04:39:21.188309 7853 consensus_queue.cc:260] T 86e243d770314d2d85cbbfc6e8264925 P 6682fed6b70747e791e1cf424b939759 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6682fed6b70747e791e1cf424b939759" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44315 } } peers { permanent_uuid: "12a3672caea9413f8a824f93100ef42c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42843 } } peers { permanent_uuid: "8a2be5f60c294266a68a6ea6147f2349" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38919 } }
I20260925 04:39:21.188654 7853 ts_tablet_manager.cc:1428] T 86e243d770314d2d85cbbfc6e8264925 P 6682fed6b70747e791e1cf424b939759: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260925 04:39:21.188910 7854 tablet_bootstrap.cc:492] T 86e243d770314d2d85cbbfc6e8264925 P 12a3672caea9413f8a824f93100ef42c: No bootstrap required, opened a new log
I20260925 04:39:21.188997 7854 ts_tablet_manager.cc:1397] T 86e243d770314d2d85cbbfc6e8264925 P 12a3672caea9413f8a824f93100ef42c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 04:39:21.189680 7856 tablet_bootstrap.cc:492] T 86e243d770314d2d85cbbfc6e8264925 P 8a2be5f60c294266a68a6ea6147f2349: No bootstrap required, opened a new log
I20260925 04:39:21.189767 7856 ts_tablet_manager.cc:1397] T 86e243d770314d2d85cbbfc6e8264925 P 8a2be5f60c294266a68a6ea6147f2349: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 04:39:21.189836 7854 raft_consensus.cc:348] T 86e243d770314d2d85cbbfc6e8264925 P 12a3672caea9413f8a824f93100ef42c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6682fed6b70747e791e1cf424b939759" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44315 } } peers { permanent_uuid: "12a3672caea9413f8a824f93100ef42c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42843 } } peers { permanent_uuid: "8a2be5f60c294266a68a6ea6147f2349" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38919 } }
I20260925 04:39:21.189944 7854 raft_consensus.cc:374] T 86e243d770314d2d85cbbfc6e8264925 P 12a3672caea9413f8a824f93100ef42c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:39:21.189972 7854 raft_consensus.cc:729] T 86e243d770314d2d85cbbfc6e8264925 P 12a3672caea9413f8a824f93100ef42c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12a3672caea9413f8a824f93100ef42c, State: Initialized, Role: FOLLOWER
I20260925 04:39:21.190104 7854 consensus_queue.cc:260] T 86e243d770314d2d85cbbfc6e8264925 P 12a3672caea9413f8a824f93100ef42c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6682fed6b70747e791e1cf424b939759" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44315 } } peers { permanent_uuid: "12a3672caea9413f8a824f93100ef42c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42843 } } peers { permanent_uuid: "8a2be5f60c294266a68a6ea6147f2349" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38919 } }
I20260925 04:39:21.190222 7856 raft_consensus.cc:348] T 86e243d770314d2d85cbbfc6e8264925 P 8a2be5f60c294266a68a6ea6147f2349 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6682fed6b70747e791e1cf424b939759" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44315 } } peers { permanent_uuid: "12a3672caea9413f8a824f93100ef42c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42843 } } peers { permanent_uuid: "8a2be5f60c294266a68a6ea6147f2349" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38919 } }
I20260925 04:39:21.190318 7856 raft_consensus.cc:374] T 86e243d770314d2d85cbbfc6e8264925 P 8a2be5f60c294266a68a6ea6147f2349 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:39:21.190347 7856 raft_consensus.cc:729] T 86e243d770314d2d85cbbfc6e8264925 P 8a2be5f60c294266a68a6ea6147f2349 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a2be5f60c294266a68a6ea6147f2349, State: Initialized, Role: FOLLOWER
I20260925 04:39:21.190424 7854 ts_tablet_manager.cc:1428] T 86e243d770314d2d85cbbfc6e8264925 P 12a3672caea9413f8a824f93100ef42c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 04:39:21.190466 7856 consensus_queue.cc:260] T 86e243d770314d2d85cbbfc6e8264925 P 8a2be5f60c294266a68a6ea6147f2349 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6682fed6b70747e791e1cf424b939759" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44315 } } peers { permanent_uuid: "12a3672caea9413f8a824f93100ef42c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42843 } } peers { permanent_uuid: "8a2be5f60c294266a68a6ea6147f2349" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38919 } }
I20260925 04:39:21.190766 7856 ts_tablet_manager.cc:1428] T 86e243d770314d2d85cbbfc6e8264925 P 8a2be5f60c294266a68a6ea6147f2349: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 04:39:21.191434 7760 heartbeater.cc:503] Master 127.0.98.60:35759 was elected leader, sending a full tablet report...
I20260925 04:39:21.191704 7863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:21.191228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41387 (local address 127.0.98.2:54167)
0925 04:39:21.191505 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:21.191571 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":143,"threads_started":1}
I20260925 04:39:21.192348 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:21.191958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41387 (local address 127.0.98.1:38999)
0925 04:39:21.192200 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:21.192252 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":79,"threads_started":1}
I20260925 04:39:21.194123 7487 heartbeater.cc:503] Master 127.0.98.60:35759 was elected leader, sending a full tablet report...
I20260925 04:39:21.194224 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:21.189312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41387 (local address 127.0.98.3:40717)
0925 04:39:21.194055 (+ 4743us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:21.194115 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4669,"thread_start_us":95,"threads_started":1}
I20260925 04:39:21.194573 7624 heartbeater.cc:503] Master 127.0.98.60:35759 was elected leader, sending a full tablet report...
I20260925 04:39:21.194911 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:21.194667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41387 (local address 127.0.98.3:50995)
0925 04:39:21.194805 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:21.194852 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260925 04:39:21.195688 7863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:21.195387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41387 (local address 127.0.98.2:52245)
0925 04:39:21.195528 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:21.195582 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260925 04:39:21.196015 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:21.194186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41387 (local address 127.0.98.1:33745)
0925 04:39:21.195680 (+ 1494us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:21.195944 (+ 264us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260925 04:39:21.419126 7862 raft_consensus.cc:482] T 86e243d770314d2d85cbbfc6e8264925 P 8a2be5f60c294266a68a6ea6147f2349 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:39:21.419278 7862 raft_consensus.cc:504] T 86e243d770314d2d85cbbfc6e8264925 P 8a2be5f60c294266a68a6ea6147f2349 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6682fed6b70747e791e1cf424b939759" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44315 } } peers { permanent_uuid: "12a3672caea9413f8a824f93100ef42c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42843 } } peers { permanent_uuid: "8a2be5f60c294266a68a6ea6147f2349" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38919 } }
I20260925 04:39:21.420377 7862 leader_election.cc:288] T 86e243d770314d2d85cbbfc6e8264925 P 8a2be5f60c294266a68a6ea6147f2349 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6682fed6b70747e791e1cf424b939759 (127.0.98.3:44315), 12a3672caea9413f8a824f93100ef42c (127.0.98.2:42843)
I20260925 04:39:21.424311 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:21.420733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41227 (local address 127.0.98.2:42843)
0925 04:39:21.420851 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:21.420856 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:21.421066 (+ 210us) server_negotiation.cc:407] Connection header received
0925 04:39:21.421234 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:21.421236 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:21.421283 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:21.421340 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:39:21.421696 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:21.422199 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:21.423266 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:21.423720 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:21.423819 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:21.423884 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:21.424107 (+ 223us) server_negotiation.cc:299] Negotiation successful
0925 04:39:21.424179 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 04:39:21.424314 7849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:21.420719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60089 (local address 127.0.98.3:44315)
0925 04:39:21.420848 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:21.420852 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:21.421080 (+ 228us) server_negotiation.cc:407] Connection header received
0925 04:39:21.421225 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:21.421229 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:21.421274 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:21.421334 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:39:21.421644 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:21.422145 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:21.423263 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:21.423731 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:21.423821 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:21.423883 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:21.424120 (+ 237us) server_negotiation.cc:299] Negotiation successful
0925 04:39:21.424194 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 04:39:21.424311 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:21.420645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42843 (local address 127.0.98.1:41227)
0925 04:39:21.420973 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:21.420985 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:21.421117 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:21.421358 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:21.421361 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:21.421383 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:39:21.421540 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:21.421546 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:21.422308 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:21.422312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:21.423149 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:21.423159 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:21.423833 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:21.423837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:21.423984 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:21.424012 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 04:39:21.424077 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 04:39:21.424156 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":59,"spinlock_wait_cycles":22272}
I20260925 04:39:21.424341 7865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:21.420646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44315 (local address 127.0.98.1:60089)
0925 04:39:21.420977 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:21.420988 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:21.421121 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:21.421357 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:21.421360 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:21.421381 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:39:21.421544 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:21.421549 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:21.422250 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:21.422253 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:21.423151 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:21.423160 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:21.423834 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:21.423837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:21.423987 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:21.424022 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 04:39:21.424096 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 04:39:21.424166 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"mutex_wait_us":153,"spinlock_wait_cycles":15488,"thread_start_us":125,"threads_started":1}
I20260925 04:39:21.424896 7714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86e243d770314d2d85cbbfc6e8264925" candidate_uuid: "8a2be5f60c294266a68a6ea6147f2349" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6682fed6b70747e791e1cf424b939759" is_pre_election: true
I20260925 04:39:21.425011 7578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86e243d770314d2d85cbbfc6e8264925" candidate_uuid: "8a2be5f60c294266a68a6ea6147f2349" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12a3672caea9413f8a824f93100ef42c" is_pre_election: true
I20260925 04:39:21.425066 7714 raft_consensus.cc:2454] T 86e243d770314d2d85cbbfc6e8264925 P 6682fed6b70747e791e1cf424b939759 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a2be5f60c294266a68a6ea6147f2349 in term 0.
I20260925 04:39:21.425237 7578 raft_consensus.cc:2454] T 86e243d770314d2d85cbbfc6e8264925 P 12a3672caea9413f8a824f93100ef42c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a2be5f60c294266a68a6ea6147f2349 in term 0.
I20260925 04:39:21.425470 7377 leader_election.cc:302] T 86e243d770314d2d85cbbfc6e8264925 P 8a2be5f60c294266a68a6ea6147f2349 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6682fed6b70747e791e1cf424b939759, 8a2be5f60c294266a68a6ea6147f2349; no voters:
I20260925 04:39:21.425777 7862 raft_consensus.cc:2781] T 86e243d770314d2d85cbbfc6e8264925 P 8a2be5f60c294266a68a6ea6147f2349 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:39:21.425830 7862 raft_consensus.cc:482] T 86e243d770314d2d85cbbfc6e8264925 P 8a2be5f60c294266a68a6ea6147f2349 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:39:21.425856 7862 raft_consensus.cc:3037] T 86e243d770314d2d85cbbfc6e8264925 P 8a2be5f60c294266a68a6ea6147f2349 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:39:21.426877 7862 raft_consensus.cc:504] T 86e243d770314d2d85cbbfc6e8264925 P 8a2be5f60c294266a68a6ea6147f2349 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6682fed6b70747e791e1cf424b939759" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44315 } } peers { permanent_uuid: "12a3672caea9413f8a824f93100ef42c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42843 } } peers { permanent_uuid: "8a2be5f60c294266a68a6ea6147f2349" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38919 } }
I20260925 04:39:21.427229 7862 leader_election.cc:288] T 86e243d770314d2d85cbbfc6e8264925 P 8a2be5f60c294266a68a6ea6147f2349 [CANDIDATE]: Term 1 election: Requested vote from peers 6682fed6b70747e791e1cf424b939759 (127.0.98.3:44315), 12a3672caea9413f8a824f93100ef42c (127.0.98.2:42843)
I20260925 04:39:21.427585 7714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86e243d770314d2d85cbbfc6e8264925" candidate_uuid: "8a2be5f60c294266a68a6ea6147f2349" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6682fed6b70747e791e1cf424b939759"
I20260925 04:39:21.427644 7578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86e243d770314d2d85cbbfc6e8264925" candidate_uuid: "8a2be5f60c294266a68a6ea6147f2349" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12a3672caea9413f8a824f93100ef42c"
I20260925 04:39:21.427731 7714 raft_consensus.cc:3037] T 86e243d770314d2d85cbbfc6e8264925 P 6682fed6b70747e791e1cf424b939759 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:39:21.427755 7578 raft_consensus.cc:3037] T 86e243d770314d2d85cbbfc6e8264925 P 12a3672caea9413f8a824f93100ef42c [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:39:21.428632 7714 raft_consensus.cc:2454] T 86e243d770314d2d85cbbfc6e8264925 P 6682fed6b70747e791e1cf424b939759 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a2be5f60c294266a68a6ea6147f2349 in term 1.
I20260925 04:39:21.428645 7578 raft_consensus.cc:2454] T 86e243d770314d2d85cbbfc6e8264925 P 12a3672caea9413f8a824f93100ef42c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a2be5f60c294266a68a6ea6147f2349 in term 1.
I20260925 04:39:21.428917 7376 leader_election.cc:302] T 86e243d770314d2d85cbbfc6e8264925 P 8a2be5f60c294266a68a6ea6147f2349 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12a3672caea9413f8a824f93100ef42c, 8a2be5f60c294266a68a6ea6147f2349; no voters:
I20260925 04:39:21.429088 7862 raft_consensus.cc:2781] T 86e243d770314d2d85cbbfc6e8264925 P 8a2be5f60c294266a68a6ea6147f2349 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:39:21.429306 7862 raft_consensus.cc:686] T 86e243d770314d2d85cbbfc6e8264925 P 8a2be5f60c294266a68a6ea6147f2349 [term 1 LEADER]: Becoming Leader. State: Replica: 8a2be5f60c294266a68a6ea6147f2349, State: Running, Role: LEADER
I20260925 04:39:21.429639 7862 consensus_queue.cc:237] T 86e243d770314d2d85cbbfc6e8264925 P 8a2be5f60c294266a68a6ea6147f2349 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6682fed6b70747e791e1cf424b939759" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44315 } } peers { permanent_uuid: "12a3672caea9413f8a824f93100ef42c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42843 } } peers { permanent_uuid: "8a2be5f60c294266a68a6ea6147f2349" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38919 } }
I20260925 04:39:21.430003 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:21.429777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41387 (local address 127.0.98.1:53171)
0925 04:39:21.429882 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:21.429938 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 04:39:21.432080 7307 catalog_manager.cc:5495] T 86e243d770314d2d85cbbfc6e8264925 P 8a2be5f60c294266a68a6ea6147f2349 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a2be5f60c294266a68a6ea6147f2349 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "8a2be5f60c294266a68a6ea6147f2349" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6682fed6b70747e791e1cf424b939759" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12a3672caea9413f8a824f93100ef42c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a2be5f60c294266a68a6ea6147f2349" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38919 } health_report { overall_health: HEALTHY } } }
I20260925 04:39:21.478073 7850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:21.474677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47244 (local address 127.0.98.1:38919)
0925 04:39:21.474822 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:21.474826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:21.474839 (+ 13us) server_negotiation.cc:407] Connection header received
0925 04:39:21.474910 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:21.474913 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:21.474966 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:21.475062 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:39:21.475299 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:21.475855 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:21.476671 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:21.476904 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:21.476995 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:21.477252 (+ 257us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:39:21.477698 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:39:21.477803 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:21.477873 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 04:39:21.477931 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 04:39:21.480931 7714 raft_consensus.cc:1261] T 86e243d770314d2d85cbbfc6e8264925 P 6682fed6b70747e791e1cf424b939759 [term 1 FOLLOWER]: Refusing update from remote peer 8a2be5f60c294266a68a6ea6147f2349: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:39:21.481112 7578 raft_consensus.cc:1261] T 86e243d770314d2d85cbbfc6e8264925 P 12a3672caea9413f8a824f93100ef42c [term 1 FOLLOWER]: Refusing update from remote peer 8a2be5f60c294266a68a6ea6147f2349: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:39:21.481482 7866 consensus_queue.cc:1035] T 86e243d770314d2d85cbbfc6e8264925 P 8a2be5f60c294266a68a6ea6147f2349 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6682fed6b70747e791e1cf424b939759" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:39:21.481690 7862 consensus_queue.cc:1035] T 86e243d770314d2d85cbbfc6e8264925 P 8a2be5f60c294266a68a6ea6147f2349 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12a3672caea9413f8a824f93100ef42c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:39:21.483665 7863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:21.483304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41387 (local address 127.0.98.2:59447)
0925 04:39:21.483470 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:21.483539 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260925 04:39:21.484113 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:21.483891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41387 (local address 127.0.98.3:35881)
0925 04:39:21.483990 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:21.484043 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 04:39:21.487108 7863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:21.486838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41387 (local address 127.0.98.2:48939)
0925 04:39:21.486972 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:21.487027 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260925 04:39:21.487406 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:21.487011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41387 (local address 127.0.98.3:47307)
0925 04:39:21.487284 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:21.487337 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207}
I20260925 04:39:21.487972 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:21.487733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41387 (local address 127.0.98.1:42711)
0925 04:39:21.487845 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:21.487900 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 04:39:21.496930 7308 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:39:21.497155 7308 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:39:21.501956 7308 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d2eabe017416488d9f0a65796832ae84: Sending DeleteTablet for 3 replicas of tablet 86e243d770314d2d85cbbfc6e8264925
I20260925 04:39:21.502687 7694 tablet_service.cc:1514] Processing DeleteTablet for tablet 86e243d770314d2d85cbbfc6e8264925 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:39:21 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:44116
I20260925 04:39:21.502820 7557 tablet_service.cc:1514] Processing DeleteTablet for tablet 86e243d770314d2d85cbbfc6e8264925 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:39:21 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:47470
I20260925 04:39:21.502940 7421 tablet_service.cc:1514] Processing DeleteTablet for tablet 86e243d770314d2d85cbbfc6e8264925 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:39:21 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:47228
I20260925 04:39:21.503213 7879 tablet_replica.cc:330] T 86e243d770314d2d85cbbfc6e8264925 P 6682fed6b70747e791e1cf424b939759: stopping tablet replica
I20260925 04:39:21.503367 7879 raft_consensus.cc:2229] T 86e243d770314d2d85cbbfc6e8264925 P 6682fed6b70747e791e1cf424b939759 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:39:21.503510 7879 raft_consensus.cc:2258] T 86e243d770314d2d85cbbfc6e8264925 P 6682fed6b70747e791e1cf424b939759 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:39:21.503530 7880 tablet_replica.cc:330] T 86e243d770314d2d85cbbfc6e8264925 P 12a3672caea9413f8a824f93100ef42c: stopping tablet replica
I20260925 04:39:21.503858 7880 raft_consensus.cc:2229] T 86e243d770314d2d85cbbfc6e8264925 P 12a3672caea9413f8a824f93100ef42c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:39:21.504158 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 7270
I20260925 04:39:21.504045 7880 raft_consensus.cc:2258] T 86e243d770314d2d85cbbfc6e8264925 P 12a3672caea9413f8a824f93100ef42c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:39:21.504704 7879 ts_tablet_manager.cc:1905] T 86e243d770314d2d85cbbfc6e8264925 P 6682fed6b70747e791e1cf424b939759: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:39:21.506026 7880 ts_tablet_manager.cc:1905] T 86e243d770314d2d85cbbfc6e8264925 P 12a3672caea9413f8a824f93100ef42c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:39:21.507810 7879 ts_tablet_manager.cc:1918] T 86e243d770314d2d85cbbfc6e8264925 P 6682fed6b70747e791e1cf424b939759: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:39:21.507905 7879 log.cc:1196] T 86e243d770314d2d85cbbfc6e8264925 P 6682fed6b70747e791e1cf424b939759: Deleting WAL directory at /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-2/wal/wals/86e243d770314d2d85cbbfc6e8264925
I20260925 04:39:21.508165 7879 ts_tablet_manager.cc:1939] T 86e243d770314d2d85cbbfc6e8264925 P 6682fed6b70747e791e1cf424b939759: Deleting consensus metadata
I20260925 04:39:21.508963 7880 ts_tablet_manager.cc:1918] T 86e243d770314d2d85cbbfc6e8264925 P 12a3672caea9413f8a824f93100ef42c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:39:21.509049 7880 log.cc:1196] T 86e243d770314d2d85cbbfc6e8264925 P 12a3672caea9413f8a824f93100ef42c: Deleting WAL directory at /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311123478858-392-0/minicluster-data/ts-1/wal/wals/86e243d770314d2d85cbbfc6e8264925
I20260925 04:39:21.509047 7295 catalog_manager.cc:4841] TS 6682fed6b70747e791e1cf424b939759 (127.0.98.3:44315): tablet 86e243d770314d2d85cbbfc6e8264925 (table test-table [id=f31962847017453bb1b95f62b52154c9]) successfully deleted
I20260925 04:39:21.509321 7880 ts_tablet_manager.cc:1939] T 86e243d770314d2d85cbbfc6e8264925 P 12a3672caea9413f8a824f93100ef42c: Deleting consensus metadata
I20260925 04:39:21.510646 7294 catalog_manager.cc:4841] TS 12a3672caea9413f8a824f93100ef42c (127.0.98.2:42843): tablet 86e243d770314d2d85cbbfc6e8264925 (table test-table [id=f31962847017453bb1b95f62b52154c9]) successfully deleted
I20260925 04:39:21.512348 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 7495
I20260925 04:39:21.512528 7851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:21.512253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38919 (local address 127.0.0.1:47250)
0925 04:39:21.512388 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:21.512442 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 04:39:21.513170 7292 catalog_manager.cc:4584] TS 8a2be5f60c294266a68a6ea6147f2349 (127.0.98.1:38919): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 86e243d770314d2d85cbbfc6e8264925: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38919: connect: Connection refused (error 111)
I20260925 04:39:21.518999 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 7631
I20260925 04:39:21.524435 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 6989
I20260925 04:39:21.529663 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 7092
2026-09-25T04:39:21Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5612 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:39:21 dist-test-slave-dc95 krb5kdc[7884](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:39:21 dist-test-slave-dc95 krb5kdc[7884](info): set up 2 sockets
Sep 25 04:39:21 dist-test-slave-dc95 krb5kdc[7884](info): commencing operation
krb5kdc: starting...
W20260925 04:39:22.496109 392 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/krb5kdc/test-user.keytab.
Sep 25 04:39:22 dist-test-slave-dc95 krb5kdc[7884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:39:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:39:22Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260925 04:39:22.620105 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40557
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40557
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:39:22.733877 7897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:39:22.734186 7897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:39:22.734229 7897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:39:22.738597 7897 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:39:22.738698 7897 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 04:39:22.738716 7897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:39:22.738734 7897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:39:22.738762 7897 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:39:22.738778 7897 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:39:22.744451 7897 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40557
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:39:22.745610 7897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:39:22.746660 7897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:39:22.755203 7903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:22.755192 7905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:22.755198 7902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:22.755578 7897 server_base.cc:1034] running on GCE node
I20260925 04:39:22.756176 7897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:39:22.757328 7897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:39:22.758508 7897 hybrid_clock.cc:648] HybridClock initialized: now 1790311162758487 us; error 38 us; skew 500 ppm
Sep 25 04:39:22 dist-test-slave-dc95 krb5kdc[7884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311162, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:39:22.768608 7897 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260925 04:39:22.769644 7897 webserver.cc:460] Webserver started at http://127.0.98.62:34619/ using document root <none> and password file <none>
I20260925 04:39:22.770247 7897 fs_manager.cc:359] Metadata directory not provided
I20260925 04:39:22.770320 7897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:39:22.770539 7897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:39:22.772769 7897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/master-0/data/instance:
uuid: "bf7d385c164047fd9a6fc095948982f6"
format_stamp: "Formatted at 2026-09-25 04:39:22 on dist-test-slave-dc95"
server_key: "1d393dcf2c6e2a8b99bb572ce5962bb4"
server_key_iv: "662f7ab2316779d4a5fbbf6f7b85bd68"
server_key_version: "encryptionkey@0"
I20260925 04:39:22.773325 7897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/master-0/wal/instance:
uuid: "bf7d385c164047fd9a6fc095948982f6"
format_stamp: "Formatted at 2026-09-25 04:39:22 on dist-test-slave-dc95"
server_key: "1d393dcf2c6e2a8b99bb572ce5962bb4"
server_key_iv: "662f7ab2316779d4a5fbbf6f7b85bd68"
server_key_version: "encryptionkey@0"
I20260925 04:39:22.777796 7897 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260925 04:39:22.780441 7912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:22.781796 7897 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 04:39:22.781922 7897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/master-0/wal
uuid: "bf7d385c164047fd9a6fc095948982f6"
format_stamp: "Formatted at 2026-09-25 04:39:22 on dist-test-slave-dc95"
server_key: "1d393dcf2c6e2a8b99bb572ce5962bb4"
server_key_iv: "662f7ab2316779d4a5fbbf6f7b85bd68"
server_key_version: "encryptionkey@0"
I20260925 04:39:22.782028 7897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:39:22.805086 7897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:39:22.808442 7897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:39:22.808607 7897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:39:22.816488 7897 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40557
I20260925 04:39:22.816543 7974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40557 every 8 connection(s)
I20260925 04:39:22.817569 7897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
I20260925 04:39:22.820899 7975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:39:22.827095 7975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf7d385c164047fd9a6fc095948982f6: Bootstrap starting.
I20260925 04:39:22.827142 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 7897
I20260925 04:39:22.827234 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/master-0/wal/instance
I20260925 04:39:22.827523 392 external_mini_cluster.cc:1428] Setting key 371317e5064400a1b3917d06cfbc019e
I20260925 04:39:22.829747 7975 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf7d385c164047fd9a6fc095948982f6: Neither blocks nor log segments found. Creating new log.
I20260925 04:39:22.830511 7975 log.cc:824] T 00000000000000000000000000000000 P bf7d385c164047fd9a6fc095948982f6: Log is configured to *not* fsync() on all Append() calls
I20260925 04:39:22.832686 7975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf7d385c164047fd9a6fc095948982f6: No bootstrap required, opened a new log
Sep 25 04:39:22 dist-test-slave-dc95 krb5kdc[7884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 04:39:22.836711 7975 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf7d385c164047fd9a6fc095948982f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf7d385c164047fd9a6fc095948982f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40557 } }
I20260925 04:39:22.836992 7975 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf7d385c164047fd9a6fc095948982f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:39:22.837041 7975 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf7d385c164047fd9a6fc095948982f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf7d385c164047fd9a6fc095948982f6, State: Initialized, Role: FOLLOWER
I20260925 04:39:22.837541 7975 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf7d385c164047fd9a6fc095948982f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf7d385c164047fd9a6fc095948982f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40557 } }
I20260925 04:39:22.837695 7975 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf7d385c164047fd9a6fc095948982f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:39:22.837760 7975 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf7d385c164047fd9a6fc095948982f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:39:22.837860 7975 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf7d385c164047fd9a6fc095948982f6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:39:22.838752 7975 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf7d385c164047fd9a6fc095948982f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf7d385c164047fd9a6fc095948982f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40557 } }
I20260925 04:39:22.839103 7975 leader_election.cc:302] T 00000000000000000000000000000000 P bf7d385c164047fd9a6fc095948982f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf7d385c164047fd9a6fc095948982f6; no voters:
I20260925 04:39:22.839428 7975 leader_election.cc:288] T 00000000000000000000000000000000 P bf7d385c164047fd9a6fc095948982f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:39:22.839521 7980 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf7d385c164047fd9a6fc095948982f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:39:22.839716 7980 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf7d385c164047fd9a6fc095948982f6 [term 1 LEADER]: Becoming Leader. State: Replica: bf7d385c164047fd9a6fc095948982f6, State: Running, Role: LEADER
I20260925 04:39:22.840019 7980 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf7d385c164047fd9a6fc095948982f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf7d385c164047fd9a6fc095948982f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40557 } }
I20260925 04:39:22.840771 7975 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf7d385c164047fd9a6fc095948982f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:39:22.842159 7978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:22.828861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56438 (local address 127.0.98.62:40557)
0925 04:39:22.829308 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:22.829324 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:22.829363 (+ 39us) server_negotiation.cc:407] Connection header received
0925 04:39:22.830085 (+ 722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:22.830104 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:22.830491 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:22.830822 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:22.831636 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:22.832686 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:22.833579 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:22.834090 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:22.834242 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:22.836975 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:22.836982 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:22.837002 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:22.837043 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:22.838943 (+ 1900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:22.839418 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:22.839425 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:22.839431 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:22.839587 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:22.839977 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:22.839981 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:22.839983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:22.840486 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:22.840671 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:22.840961 (+ 290us) server_negotiation.cc:299] Negotiation successful
0925 04:39:22.841265 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":145,"threads_started":1}
I20260925 04:39:22.843006 7982 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf7d385c164047fd9a6fc095948982f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf7d385c164047fd9a6fc095948982f6. Latest consensus state: current_term: 1 leader_uuid: "bf7d385c164047fd9a6fc095948982f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf7d385c164047fd9a6fc095948982f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40557 } } }
I20260925 04:39:22.843240 7982 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf7d385c164047fd9a6fc095948982f6 [sys.catalog]: This master's current role is: LEADER
I20260925 04:39:22.843502 7981 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf7d385c164047fd9a6fc095948982f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf7d385c164047fd9a6fc095948982f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf7d385c164047fd9a6fc095948982f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40557 } } }
I20260925 04:39:22.843588 7981 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf7d385c164047fd9a6fc095948982f6 [sys.catalog]: This master's current role is: LEADER
W20260925 04:39:22.845503 7994 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bf7d385c164047fd9a6fc095948982f6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 04:39:22.845573 7994 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 04:39:22.845628 7995 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:39:22.846376 7995 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:39:22.852626 7995 catalog_manager.cc:1347] Generated new cluster ID: 43ea0a349cc04cafbb0348ff8818f7b7
I20260925 04:39:22.852716 7995 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:39:22.858181 7995 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:39:22.859526 7995 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:39:22.870497 7995 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf7d385c164047fd9a6fc095948982f6: Generated new TSK 0
I20260925 04:39:22.871395 7995 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260925 04:39:22.902622 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40557
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:39:23.017271 8001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:39:23.017577 8001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:39:23.017624 8001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:39:23.021867 8001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:39:23.021955 8001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:39:23.022004 8001 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 04:39:23.022050 8001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 04:39:23.027549 8001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:40557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8001
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:39:23.028741 8001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:39:23.029767 8001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:39:23.039386 8006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:23.039404 8007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:23.039353 8009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:23.039508 8001 server_base.cc:1034] running on GCE node
I20260925 04:39:23.039950 8001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:39:23.040598 8001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:39:23.041746 8001 hybrid_clock.cc:648] HybridClock initialized: now 1790311163041722 us; error 35 us; skew 500 ppm
Sep 25 04:39:23 dist-test-slave-dc95 krb5kdc[7884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311163, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:39:23.044616 8001 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260925 04:39:23.045542 8001 webserver.cc:460] Webserver started at http://127.0.98.1:45963/ using document root <none> and password file <none>
I20260925 04:39:23.046080 8001 fs_manager.cc:359] Metadata directory not provided
I20260925 04:39:23.046147 8001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:39:23.046351 8001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:39:23.048681 8001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/ts-0/data/instance:
uuid: "d73297192af149f3b757984ddd9afcb0"
format_stamp: "Formatted at 2026-09-25 04:39:23 on dist-test-slave-dc95"
server_key: "716ad856a82e718de732ecb764d7ef17"
server_key_iv: "2d1822dd4e632b55f338e59c7996696a"
server_key_version: "encryptionkey@0"
I20260925 04:39:23.049225 8001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d73297192af149f3b757984ddd9afcb0"
format_stamp: "Formatted at 2026-09-25 04:39:23 on dist-test-slave-dc95"
server_key: "716ad856a82e718de732ecb764d7ef17"
server_key_iv: "2d1822dd4e632b55f338e59c7996696a"
server_key_version: "encryptionkey@0"
I20260925 04:39:23.052968 8001 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:39:23.055537 8016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:23.057039 8001 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:39:23.057191 8001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/ts-0/wal
uuid: "d73297192af149f3b757984ddd9afcb0"
format_stamp: "Formatted at 2026-09-25 04:39:23 on dist-test-slave-dc95"
server_key: "716ad856a82e718de732ecb764d7ef17"
server_key_iv: "2d1822dd4e632b55f338e59c7996696a"
server_key_version: "encryptionkey@0"
I20260925 04:39:23.057327 8001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:39:23.076953 8001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:39:23.080050 8001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:39:23.080243 8001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:39:23.081737 8001 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:39:23.081794 8001 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:23.081848 8001 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:39:23.081884 8001 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:23.091540 8001 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33961
I20260925 04:39:23.091683 8131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33961 every 8 connection(s)
Sep 25 04:39:23 dist-test-slave-dc95 krb5kdc[7884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311163, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 04:39:23.092526 8001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
I20260925 04:39:23.096364 7978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:23.082736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34943 (local address 127.0.98.62:40557)
0925 04:39:23.082881 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:23.082886 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:23.083823 (+ 937us) server_negotiation.cc:407] Connection header received
0925 04:39:23.085113 (+ 1290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:23.085117 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:23.085181 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:23.085240 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:23.087148 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:23.088018 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:23.089442 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:23.089796 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:23.089929 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:23.092675 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:23.092680 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:23.092683 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:23.092714 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:23.094272 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:23.094750 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:23.094752 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:23.094754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:23.094834 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:23.095145 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:23.095147 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:23.095148 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:23.095334 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:23.095413 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:23.096100 (+ 687us) server_negotiation.cc:299] Negotiation successful
0925 04:39:23.096230 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 04:39:23.097313 8025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:23.083125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40557 (local address 127.0.98.1:34943)
0925 04:39:23.083648 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:23.083703 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:23.084830 (+ 1127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:23.085670 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:23.085679 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:23.086194 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:23.086946 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:23.086971 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:23.088168 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:23.088172 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:23.089301 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:23.089313 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:23.090009 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:23.090015 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:23.090222 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:23.090868 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:23.090894 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:23.092478 (+ 1584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:39:23.094391 (+ 1913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:23.094405 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:23.094417 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:23.094670 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:23.094934 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:23.094936 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:23.094938 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:23.095076 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:23.095542 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:39:23.095548 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:39:23.095865 (+ 317us) client_negotiation.cc:769] Sending connection context
0925 04:39:23.096040 (+ 175us) client_negotiation.cc:240] Negotiation successful
0925 04:39:23.096322 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":111,"threads_started":1}
I20260925 04:39:23.098830 8132 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40557
I20260925 04:39:23.099000 8132 heartbeater.cc:464] Registering TS with master...
I20260925 04:39:23.099411 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 8001
I20260925 04:39:23.099524 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/ts-0/wal/instance
I20260925 04:39:23.099686 8132 heartbeater.cc:511] Master 127.0.98.62:40557 requested a full tablet report, sending...
I20260925 04:39:23.099908 392 external_mini_cluster.cc:1428] Setting key 5b40f27c82045ba7cd18c69d4efdc53d
I20260925 04:39:23.101281 7928 ts_manager.cc:194] Registered new tserver with Master: d73297192af149f3b757984ddd9afcb0 (127.0.98.1:33961)
I20260925 04:39:23.102440 7928 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34943
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 04:39:23.110731 8138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:23.101246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40557 (local address 127.0.98.1:48605)
0925 04:39:23.104409 (+ 3163us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:23.104423 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:23.104522 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:23.104744 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:23.104747 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:23.104938 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:23.105036 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:23.105041 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:23.105769 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:23.105772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:23.106349 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:23.106359 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:23.106784 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:23.106788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:23.106878 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:23.107348 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:23.107360 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:23.107731 (+ 371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:39:23.109301 (+ 1570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:23.109304 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:23.109306 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:23.109500 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:23.109749 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:23.109752 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:23.109754 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:23.109831 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:23.110273 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:39:23.110275 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:39:23.110361 (+ 86us) client_negotiation.cc:769] Sending connection context
0925 04:39:23.110455 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 04:39:23.110579 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3079,"thread_start_us":108,"threads_started":1}
I20260925 04:39:23.110781 7978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:23.101608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48605 (local address 127.0.98.62:40557)
0925 04:39:23.101716 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:23.101720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:23.104497 (+ 2777us) server_negotiation.cc:407] Connection header received
0925 04:39:23.104606 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:23.104609 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:23.104652 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:23.104713 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:23.105138 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:23.105653 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:23.106462 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:23.106685 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:23.106767 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:23.107854 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:23.107858 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:23.107861 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:23.107885 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:23.109200 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:23.109587 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:23.109591 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:23.109593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:23.109670 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:23.109934 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:23.109938 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:23.109940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:23.110175 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:23.110266 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:23.110549 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 04:39:23.110662 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260925 04:39:23.128906 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40557
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:39:23.240703 8141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:39:23.241014 8141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:39:23.241070 8141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:39:23.245362 8141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:39:23.245435 8141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:39:23.245478 8141 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 04:39:23.245512 8141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 04:39:23.250938 8141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:40557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:39:23.252043 8141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:39:23.253062 8141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:39:23.262838 8146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:23.262835 8147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:23.262835 8149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:23.262941 8141 server_base.cc:1034] running on GCE node
I20260925 04:39:23.263530 8141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:39:23.264245 8141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:39:23.265442 8141 hybrid_clock.cc:648] HybridClock initialized: now 1790311163265419 us; error 36 us; skew 500 ppm
Sep 25 04:39:23 dist-test-slave-dc95 krb5kdc[7884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311163, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:39:23.268229 8141 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260925 04:39:23.269163 8141 webserver.cc:460] Webserver started at http://127.0.98.2:43039/ using document root <none> and password file <none>
I20260925 04:39:23.269696 8141 fs_manager.cc:359] Metadata directory not provided
I20260925 04:39:23.269752 8141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:39:23.269919 8141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:39:23.271984 8141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/ts-1/data/instance:
uuid: "bf756d2a72414257ae7197d5ca26d0d7"
format_stamp: "Formatted at 2026-09-25 04:39:23 on dist-test-slave-dc95"
server_key: "1b999e9b10a99750c5ab240ae986c0e7"
server_key_iv: "d42d2021b68f813221bd537e96115969"
server_key_version: "encryptionkey@0"
I20260925 04:39:23.272497 8141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/ts-1/wal/instance:
uuid: "bf756d2a72414257ae7197d5ca26d0d7"
format_stamp: "Formatted at 2026-09-25 04:39:23 on dist-test-slave-dc95"
server_key: "1b999e9b10a99750c5ab240ae986c0e7"
server_key_iv: "d42d2021b68f813221bd537e96115969"
server_key_version: "encryptionkey@0"
I20260925 04:39:23.276098 8141 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 04:39:23.278448 8156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:23.279735 8141 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:39:23.279855 8141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/ts-1/wal
uuid: "bf756d2a72414257ae7197d5ca26d0d7"
format_stamp: "Formatted at 2026-09-25 04:39:23 on dist-test-slave-dc95"
server_key: "1b999e9b10a99750c5ab240ae986c0e7"
server_key_iv: "d42d2021b68f813221bd537e96115969"
server_key_version: "encryptionkey@0"
I20260925 04:39:23.279973 8141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:39:23.307242 8141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:39:23.310479 8141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:39:23.310670 8141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:39:23.312250 8141 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:39:23.312305 8141 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:23.312345 8141 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:39:23.312362 8141 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 04:39:23 dist-test-slave-dc95 krb5kdc[7884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311163, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 04:39:23.322816 8141 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42389
I20260925 04:39:23.322863 8271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42389 every 8 connection(s)
I20260925 04:39:23.324066 8141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
I20260925 04:39:23.325205 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 8141
I20260925 04:39:23.325322 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/ts-1/wal/instance
I20260925 04:39:23.325629 392 external_mini_cluster.cc:1428] Setting key 31b3b4b13a83bd7aef810e20c3aceacd
I20260925 04:39:23.326017 7978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:23.313004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35533 (local address 127.0.98.62:40557)
0925 04:39:23.313158 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:23.313164 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:23.314091 (+ 927us) server_negotiation.cc:407] Connection header received
0925 04:39:23.315320 (+ 1229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:23.315324 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:23.315385 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:23.315481 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:23.316788 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:23.317360 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:23.318575 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:23.318849 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:23.318944 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:23.322041 (+ 3097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:23.322044 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:23.322046 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:23.322074 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:23.323691 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:23.324346 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:23.324350 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:23.324351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:23.324449 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:23.324861 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:23.324865 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:23.324867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:23.325084 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:23.325173 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:23.325725 (+ 552us) server_negotiation.cc:299] Negotiation successful
0925 04:39:23.325856 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 04:39:23.327004 8165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:23.313393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40557 (local address 127.0.98.2:35533)
0925 04:39:23.313899 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:23.313966 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:23.315059 (+ 1093us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:23.315708 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:23.315715 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:23.316074 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:23.316647 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:23.316664 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:23.317493 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:23.317496 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:23.318453 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:23.318471 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:23.319003 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:23.319008 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:23.319188 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:23.319867 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:23.319884 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:23.321807 (+ 1923us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:39:23.323858 (+ 2051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:23.323864 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:23.323881 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:23.324232 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:23.324576 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:23.324579 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:23.324581 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:23.324747 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:23.325185 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:39:23.325192 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:39:23.325498 (+ 306us) client_negotiation.cc:769] Sending connection context
0925 04:39:23.325666 (+ 168us) client_negotiation.cc:240] Negotiation successful
0925 04:39:23.326014 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":99,"threads_started":1}
I20260925 04:39:23.328346 8272 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40557
I20260925 04:39:23.328511 8272 heartbeater.cc:464] Registering TS with master...
I20260925 04:39:23.329056 8272 heartbeater.cc:511] Master 127.0.98.62:40557 requested a full tablet report, sending...
I20260925 04:39:23.330098 7929 ts_manager.cc:194] Registered new tserver with Master: bf756d2a72414257ae7197d5ca26d0d7 (127.0.98.2:42389)
I20260925 04:39:23.330813 7929 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35533
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260925 04:39:23.337850 7978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:23.330609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36537 (local address 127.0.98.62:40557)
0925 04:39:23.330718 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:23.330723 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:23.330944 (+ 221us) server_negotiation.cc:407] Connection header received
0925 04:39:23.331095 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:23.331098 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:23.331140 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:23.331201 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:23.331702 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:23.332206 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:23.332996 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:23.333198 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:23.333279 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:23.334452 (+ 1173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:23.334456 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:23.334458 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:23.334487 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:23.335990 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:23.336415 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:23.336419 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:23.336421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:23.336514 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:23.336791 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:23.336794 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:23.336796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:23.337113 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:23.337215 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:23.337564 (+ 349us) server_negotiation.cc:299] Negotiation successful
0925 04:39:23.337685 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 04:39:23.337925 8279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:23.330536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40557 (local address 127.0.98.2:36537)
0925 04:39:23.330869 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:23.330882 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:23.331010 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:23.331231 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:23.331234 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:23.331473 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:23.331587 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:23.331593 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:23.332326 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:23.332329 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:23.332895 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:23.332902 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:23.333300 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:23.333303 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:23.333398 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:23.333921 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:23.333933 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:23.334306 (+ 373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:39:23.336115 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:23.336118 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:23.336120 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:23.336322 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:23.336608 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:23.336610 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:23.336612 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:23.336693 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:23.337241 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:39:23.337243 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:39:23.337341 (+ 98us) client_negotiation.cc:769] Sending connection context
0925 04:39:23.337473 (+ 132us) client_negotiation.cc:240] Negotiation successful
0925 04:39:23.337652 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":99,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260925 04:39:23.355125 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40557
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:39:23.469981 8281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:39:23.470278 8281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:39:23.470340 8281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:39:23.474546 8281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:39:23.474637 8281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:39:23.474691 8281 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 04:39:23.474746 8281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 04:39:23.480240 8281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:40557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:39:23.481392 8281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:39:23.482441 8281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:39:23.492555 8287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:23.492573 8286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:23.492555 8289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:23.492640 8281 server_base.cc:1034] running on GCE node
I20260925 04:39:23.493257 8281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:39:23.494001 8281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:39:23.495191 8281 hybrid_clock.cc:648] HybridClock initialized: now 1790311163495167 us; error 40 us; skew 500 ppm
Sep 25 04:39:23 dist-test-slave-dc95 krb5kdc[7884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311163, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:39:23.498353 8281 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260925 04:39:23.499442 8281 webserver.cc:460] Webserver started at http://127.0.98.3:37557/ using document root <none> and password file <none>
I20260925 04:39:23.500123 8281 fs_manager.cc:359] Metadata directory not provided
I20260925 04:39:23.500195 8281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:39:23.500414 8281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:39:23.502563 8281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/ts-2/data/instance:
uuid: "a310b4a5969a4b5686535c2d19fd3c24"
format_stamp: "Formatted at 2026-09-25 04:39:23 on dist-test-slave-dc95"
server_key: "89f34a74ad04e3cb68985bb2d39847a0"
server_key_iv: "2898f501a8c789ee6048bfb2c7283bbe"
server_key_version: "encryptionkey@0"
I20260925 04:39:23.503134 8281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a310b4a5969a4b5686535c2d19fd3c24"
format_stamp: "Formatted at 2026-09-25 04:39:23 on dist-test-slave-dc95"
server_key: "89f34a74ad04e3cb68985bb2d39847a0"
server_key_iv: "2898f501a8c789ee6048bfb2c7283bbe"
server_key_version: "encryptionkey@0"
I20260925 04:39:23.507071 8281 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 04:39:23.509753 8296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:23.511034 8281 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:39:23.511155 8281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/ts-2/wal
uuid: "a310b4a5969a4b5686535c2d19fd3c24"
format_stamp: "Formatted at 2026-09-25 04:39:23 on dist-test-slave-dc95"
server_key: "89f34a74ad04e3cb68985bb2d39847a0"
server_key_iv: "2898f501a8c789ee6048bfb2c7283bbe"
server_key_version: "encryptionkey@0"
I20260925 04:39:23.511266 8281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:39:23.525918 8281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:39:23.529667 8281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:39:23.529874 8281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:39:23.531423 8281 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:39:23.531482 8281 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:23.531538 8281 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:39:23.531597 8281 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 04:39:23 dist-test-slave-dc95 krb5kdc[7884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311163, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 04:39:23.542222 8281 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46153
I20260925 04:39:23.542505 8411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46153 every 8 connection(s)
I20260925 04:39:23.543373 8281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
I20260925 04:39:23.545905 7978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:23.532279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53691 (local address 127.0.98.62:40557)
0925 04:39:23.532444 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:23.532449 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:23.533344 (+ 895us) server_negotiation.cc:407] Connection header received
0925 04:39:23.534504 (+ 1160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:23.534509 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:23.534576 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:23.534690 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:23.536506 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:23.537094 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:23.538422 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:23.538649 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:23.538750 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:23.541809 (+ 3059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:23.541813 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:23.541817 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:23.541850 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:23.543596 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:23.544301 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:23.544305 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:23.544308 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:23.544411 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:23.544715 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:23.544717 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:23.544718 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:23.544905 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:23.544977 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:23.545650 (+ 673us) server_negotiation.cc:299] Negotiation successful
0925 04:39:23.545776 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 04:39:23.546614 8305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:23.532681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40557 (local address 127.0.98.3:53691)
0925 04:39:23.533152 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:23.533205 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:23.534203 (+ 998us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:23.534942 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:23.534952 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:23.535755 (+ 803us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:23.536311 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:23.536327 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:23.537231 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:23.537235 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:23.538274 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:23.538289 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:23.538839 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:23.538844 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:23.539025 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:23.539803 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:23.539821 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:23.541537 (+ 1716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:39:23.543743 (+ 2206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:23.543749 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:23.543767 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:23.544179 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:23.544507 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:23.544509 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:23.544511 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:23.544649 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:23.545021 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:39:23.545027 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:39:23.545272 (+ 245us) client_negotiation.cc:769] Sending connection context
0925 04:39:23.545440 (+ 168us) client_negotiation.cc:240] Negotiation successful
0925 04:39:23.545727 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":101,"threads_started":1}
I20260925 04:39:23.547933 8412 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40557
I20260925 04:39:23.548084 8412 heartbeater.cc:464] Registering TS with master...
I20260925 04:39:23.548609 8412 heartbeater.cc:511] Master 127.0.98.62:40557 requested a full tablet report, sending...
I20260925 04:39:23.549690 7928 ts_manager.cc:194] Registered new tserver with Master: a310b4a5969a4b5686535c2d19fd3c24 (127.0.98.3:46153)
I20260925 04:39:23.550371 7928 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53691
I20260925 04:39:23.552160 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 8281
I20260925 04:39:23.552296 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/ts-2/wal/instance
I20260925 04:39:23.552596 392 external_mini_cluster.cc:1428] Setting key a3d9605e872ec9e142b27198f9b26d8a
I20260925 04:39:23.554749 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:39:23.557937 8418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:23.549882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40557 (local address 127.0.98.3:40523)
0925 04:39:23.550155 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:23.550167 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:23.550260 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:23.550589 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:23.550592 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:23.550797 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:23.550905 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:23.550912 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:23.551677 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:23.551680 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:23.552559 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:23.552570 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:23.553184 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:23.553188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:23.553284 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:23.553796 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:23.553811 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:23.554180 (+ 369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:39:23.556342 (+ 2162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:23.556345 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:23.556347 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:23.556619 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:23.556970 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:23.556972 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:23.556974 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:23.557082 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:23.557499 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:39:23.557501 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:39:23.557586 (+ 85us) client_negotiation.cc:769] Sending connection context
0925 04:39:23.557660 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 04:39:23.557787 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":4,"thread_start_us":102,"threads_started":1}
I20260925 04:39:23.558002 7978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:23.549920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40523 (local address 127.0.98.62:40557)
0925 04:39:23.550064 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:23.550068 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:23.550229 (+ 161us) server_negotiation.cc:407] Connection header received
0925 04:39:23.550434 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:23.550438 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:23.550497 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:23.550570 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:23.551022 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:23.551531 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:23.552703 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:23.553068 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:23.553159 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:23.554313 (+ 1154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:23.554317 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:23.554320 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:23.554358 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:23.556208 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:23.556755 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:23.556759 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:23.556761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:23.556871 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:23.557161 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:23.557164 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:23.557166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:23.557416 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:23.557518 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:23.557734 (+ 216us) server_negotiation.cc:299] Negotiation successful
0925 04:39:23.557858 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
Sep 25 04:39:23 dist-test-slave-dc95 krb5kdc[7884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311163, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 04:39:23 dist-test-slave-dc95 krb5kdc[7884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:39:23 dist-test-slave-dc95 krb5kdc[7884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 04:39:23.580107 7978 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:39:23.573606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56440 (local address 127.0.98.62:40557)
0925 04:39:23.573743 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:23.573746 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:23.573761 (+ 15us) server_negotiation.cc:407] Connection header received
0925 04:39:23.573817 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:23.573819 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:23.573861 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:23.573935 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:23.574645 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:23.575238 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:23.576158 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:23.576421 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:23.576521 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:23.579965 (+ 3444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56440: BlockingRecv error: recv got EOF from 127.0.0.1:56440 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 25 04:39:23 dist-test-slave-dc95 krb5kdc[7884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311163, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 04:39:23.590776 7978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:23.581659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56456 (local address 127.0.98.62:40557)
0925 04:39:23.581833 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:23.581837 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:23.581869 (+ 32us) server_negotiation.cc:407] Connection header received
0925 04:39:23.582143 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:23.582146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:23.582200 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:23.582273 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:23.583074 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:23.583689 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:23.584505 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:23.584734 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:23.584861 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:23.587384 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:23.587387 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:23.587390 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:23.587422 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:23.588897 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:23.589417 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:23.589421 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:23.589422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:23.589516 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:23.589856 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:23.589859 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:23.589860 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:23.590106 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:23.590218 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:23.590529 (+ 311us) server_negotiation.cc:299] Negotiation successful
0925 04:39:23.590643 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 04:39:23.594094 7928 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56456:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:39:23.597417 7928 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:39:23.612668 8436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:23.606708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46153 (local address 127.0.0.1:39064)
0925 04:39:23.607951 (+ 1243us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:23.607963 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:23.608154 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:23.608816 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:23.608818 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:23.608833 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:39:23.608939 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:23.608946 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:23.610060 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:23.610064 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:23.611099 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:23.611107 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:23.612015 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:23.612019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:23.612147 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:23.612188 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 04:39:23.612346 (+ 158us) client_negotiation.cc:240] Negotiation successful
0925 04:39:23.612497 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1183,"mutex_wait_us":54,"thread_start_us":114,"threads_started":1}
I20260925 04:39:23.612671 8438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:23.607341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39064 (local address 127.0.98.3:46153)
0925 04:39:23.607807 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:23.607819 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:23.608080 (+ 261us) server_negotiation.cc:407] Connection header received
0925 04:39:23.608296 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:23.608301 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:23.608519 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:23.608648 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:39:23.609141 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:23.609950 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:23.611215 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:23.611893 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:23.612005 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:23.612203 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:23.612391 (+ 188us) server_negotiation.cc:299] Negotiation successful
0925 04:39:23.612508 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":123,"threads_started":1}
I20260925 04:39:23.612668 8434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:23.606560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42389 (local address 127.0.0.1:49130)
0925 04:39:23.606890 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:23.606948 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:23.607075 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:23.608068 (+ 993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:23.608074 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:23.608097 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:39:23.608235 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:23.608242 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:23.609497 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:23.609500 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:23.610926 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:23.610936 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:23.612284 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:23.612288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:23.612373 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:23.612396 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 04:39:23.612452 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 04:39:23.612520 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":3,"thread_start_us":77,"threads_started":1}
I20260925 04:39:23.613272 8435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:23.606716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49130 (local address 127.0.98.2:42389)
0925 04:39:23.607324 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:23.607336 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:23.607359 (+ 23us) server_negotiation.cc:407] Connection header received
0925 04:39:23.607442 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:23.607447 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:23.607705 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:23.607870 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:39:23.608360 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:23.609372 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:23.611086 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:23.611961 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:23.612058 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:23.612224 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:23.612739 (+ 515us) server_negotiation.cc:299] Negotiation successful
0925 04:39:23.613138 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":121,"threads_started":1}
I20260925 04:39:23.613734 8437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:23.607350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33961 (local address 127.0.0.1:43978)
0925 04:39:23.608162 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:23.608174 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:23.608249 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:23.608669 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:23.608672 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:23.608691 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:39:23.608783 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:23.608788 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:23.610132 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:23.610136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:23.611753 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:23.611764 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:23.613333 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:23.613337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:23.613447 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:23.613470 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 04:39:23.613541 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 04:39:23.613610 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":758,"thread_start_us":105,"threads_started":1}
I20260925 04:39:23.614931 8439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:23.607773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43978 (local address 127.0.98.1:33961)
0925 04:39:23.608235 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:23.608242 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:23.608262 (+ 20us) server_negotiation.cc:407] Connection header received
0925 04:39:23.608327 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:23.608331 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:23.608526 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:23.608660 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:39:23.608895 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:23.609970 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:23.611886 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:23.612674 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:23.613148 (+ 474us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:23.613266 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:23.614665 (+ 1399us) server_negotiation.cc:299] Negotiation successful
0925 04:39:23.614784 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":91,"threads_started":1}
I20260925 04:39:23.616262 8346 tablet_service.cc:1467] Processing CreateTablet for tablet a90b585dac3e4eeaa4184fb2311b995b (DEFAULT_TABLE table=test-table [id=93fab460a95746f38c44945e35accf0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:39:23.616271 8206 tablet_service.cc:1467] Processing CreateTablet for tablet a90b585dac3e4eeaa4184fb2311b995b (DEFAULT_TABLE table=test-table [id=93fab460a95746f38c44945e35accf0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:39:23.617347 8346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a90b585dac3e4eeaa4184fb2311b995b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:39:23.617352 8206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a90b585dac3e4eeaa4184fb2311b995b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:39:23.617996 8066 tablet_service.cc:1467] Processing CreateTablet for tablet a90b585dac3e4eeaa4184fb2311b995b (DEFAULT_TABLE table=test-table [id=93fab460a95746f38c44945e35accf0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:39:23.619599 8066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a90b585dac3e4eeaa4184fb2311b995b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:39:23.624713 8440 tablet_bootstrap.cc:492] T a90b585dac3e4eeaa4184fb2311b995b P a310b4a5969a4b5686535c2d19fd3c24: Bootstrap starting.
I20260925 04:39:23.625851 8441 tablet_bootstrap.cc:492] T a90b585dac3e4eeaa4184fb2311b995b P bf756d2a72414257ae7197d5ca26d0d7: Bootstrap starting.
I20260925 04:39:23.626943 8440 tablet_bootstrap.cc:654] T a90b585dac3e4eeaa4184fb2311b995b P a310b4a5969a4b5686535c2d19fd3c24: Neither blocks nor log segments found. Creating new log.
I20260925 04:39:23.627731 8440 log.cc:824] T a90b585dac3e4eeaa4184fb2311b995b P a310b4a5969a4b5686535c2d19fd3c24: Log is configured to *not* fsync() on all Append() calls
I20260925 04:39:23.628099 8441 tablet_bootstrap.cc:654] T a90b585dac3e4eeaa4184fb2311b995b P bf756d2a72414257ae7197d5ca26d0d7: Neither blocks nor log segments found. Creating new log.
I20260925 04:39:23.628614 8442 tablet_bootstrap.cc:492] T a90b585dac3e4eeaa4184fb2311b995b P d73297192af149f3b757984ddd9afcb0: Bootstrap starting.
I20260925 04:39:23.628883 8441 log.cc:824] T a90b585dac3e4eeaa4184fb2311b995b P bf756d2a72414257ae7197d5ca26d0d7: Log is configured to *not* fsync() on all Append() calls
I20260925 04:39:23.630666 8442 tablet_bootstrap.cc:654] T a90b585dac3e4eeaa4184fb2311b995b P d73297192af149f3b757984ddd9afcb0: Neither blocks nor log segments found. Creating new log.
I20260925 04:39:23.630976 8441 tablet_bootstrap.cc:492] T a90b585dac3e4eeaa4184fb2311b995b P bf756d2a72414257ae7197d5ca26d0d7: No bootstrap required, opened a new log
I20260925 04:39:23.630995 8440 tablet_bootstrap.cc:492] T a90b585dac3e4eeaa4184fb2311b995b P a310b4a5969a4b5686535c2d19fd3c24: No bootstrap required, opened a new log
I20260925 04:39:23.631130 8441 ts_tablet_manager.cc:1397] T a90b585dac3e4eeaa4184fb2311b995b P bf756d2a72414257ae7197d5ca26d0d7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 04:39:23.631172 8440 ts_tablet_manager.cc:1397] T a90b585dac3e4eeaa4184fb2311b995b P a310b4a5969a4b5686535c2d19fd3c24: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 04:39:23.631506 8442 log.cc:824] T a90b585dac3e4eeaa4184fb2311b995b P d73297192af149f3b757984ddd9afcb0: Log is configured to *not* fsync() on all Append() calls
I20260925 04:39:23.633571 8442 tablet_bootstrap.cc:492] T a90b585dac3e4eeaa4184fb2311b995b P d73297192af149f3b757984ddd9afcb0: No bootstrap required, opened a new log
I20260925 04:39:23.633802 8442 ts_tablet_manager.cc:1397] T a90b585dac3e4eeaa4184fb2311b995b P d73297192af149f3b757984ddd9afcb0: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 04:39:23.634312 8441 raft_consensus.cc:348] T a90b585dac3e4eeaa4184fb2311b995b P bf756d2a72414257ae7197d5ca26d0d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf756d2a72414257ae7197d5ca26d0d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 } } peers { permanent_uuid: "a310b4a5969a4b5686535c2d19fd3c24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46153 } } peers { permanent_uuid: "d73297192af149f3b757984ddd9afcb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33961 } }
I20260925 04:39:23.634542 8441 raft_consensus.cc:374] T a90b585dac3e4eeaa4184fb2311b995b P bf756d2a72414257ae7197d5ca26d0d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:39:23.634593 8441 raft_consensus.cc:729] T a90b585dac3e4eeaa4184fb2311b995b P bf756d2a72414257ae7197d5ca26d0d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf756d2a72414257ae7197d5ca26d0d7, State: Initialized, Role: FOLLOWER
I20260925 04:39:23.634871 8440 raft_consensus.cc:348] T a90b585dac3e4eeaa4184fb2311b995b P a310b4a5969a4b5686535c2d19fd3c24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf756d2a72414257ae7197d5ca26d0d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 } } peers { permanent_uuid: "a310b4a5969a4b5686535c2d19fd3c24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46153 } } peers { permanent_uuid: "d73297192af149f3b757984ddd9afcb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33961 } }
I20260925 04:39:23.635031 8441 consensus_queue.cc:260] T a90b585dac3e4eeaa4184fb2311b995b P bf756d2a72414257ae7197d5ca26d0d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf756d2a72414257ae7197d5ca26d0d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 } } peers { permanent_uuid: "a310b4a5969a4b5686535c2d19fd3c24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46153 } } peers { permanent_uuid: "d73297192af149f3b757984ddd9afcb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33961 } }
I20260925 04:39:23.635146 8440 raft_consensus.cc:374] T a90b585dac3e4eeaa4184fb2311b995b P a310b4a5969a4b5686535c2d19fd3c24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:39:23.635203 8440 raft_consensus.cc:729] T a90b585dac3e4eeaa4184fb2311b995b P a310b4a5969a4b5686535c2d19fd3c24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a310b4a5969a4b5686535c2d19fd3c24, State: Initialized, Role: FOLLOWER
I20260925 04:39:23.635691 8272 heartbeater.cc:503] Master 127.0.98.62:40557 was elected leader, sending a full tablet report...
I20260925 04:39:23.635708 8440 consensus_queue.cc:260] T a90b585dac3e4eeaa4184fb2311b995b P a310b4a5969a4b5686535c2d19fd3c24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf756d2a72414257ae7197d5ca26d0d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 } } peers { permanent_uuid: "a310b4a5969a4b5686535c2d19fd3c24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46153 } } peers { permanent_uuid: "d73297192af149f3b757984ddd9afcb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33961 } }
I20260925 04:39:23.636757 8412 heartbeater.cc:503] Master 127.0.98.62:40557 was elected leader, sending a full tablet report...
I20260925 04:39:23.636766 8440 ts_tablet_manager.cc:1428] T a90b585dac3e4eeaa4184fb2311b995b P a310b4a5969a4b5686535c2d19fd3c24: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260925 04:39:23.638103 8442 raft_consensus.cc:348] T a90b585dac3e4eeaa4184fb2311b995b P d73297192af149f3b757984ddd9afcb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf756d2a72414257ae7197d5ca26d0d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 } } peers { permanent_uuid: "a310b4a5969a4b5686535c2d19fd3c24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46153 } } peers { permanent_uuid: "d73297192af149f3b757984ddd9afcb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33961 } }
I20260925 04:39:23.638324 8441 ts_tablet_manager.cc:1428] T a90b585dac3e4eeaa4184fb2311b995b P bf756d2a72414257ae7197d5ca26d0d7: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260925 04:39:23.638391 8442 raft_consensus.cc:374] T a90b585dac3e4eeaa4184fb2311b995b P d73297192af149f3b757984ddd9afcb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:39:23.638453 8442 raft_consensus.cc:729] T a90b585dac3e4eeaa4184fb2311b995b P d73297192af149f3b757984ddd9afcb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73297192af149f3b757984ddd9afcb0, State: Initialized, Role: FOLLOWER
I20260925 04:39:23.638878 8442 consensus_queue.cc:260] T a90b585dac3e4eeaa4184fb2311b995b P d73297192af149f3b757984ddd9afcb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf756d2a72414257ae7197d5ca26d0d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 } } peers { permanent_uuid: "a310b4a5969a4b5686535c2d19fd3c24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46153 } } peers { permanent_uuid: "d73297192af149f3b757984ddd9afcb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33961 } }
I20260925 04:39:23.639900 8442 ts_tablet_manager.cc:1428] T a90b585dac3e4eeaa4184fb2311b995b P d73297192af149f3b757984ddd9afcb0: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 04:39:23.640089 8132 heartbeater.cc:503] Master 127.0.98.62:40557 was elected leader, sending a full tablet report...
I20260925 04:39:23.664947 8447 raft_consensus.cc:482] T a90b585dac3e4eeaa4184fb2311b995b P a310b4a5969a4b5686535c2d19fd3c24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:39:23.665163 8447 raft_consensus.cc:504] T a90b585dac3e4eeaa4184fb2311b995b P a310b4a5969a4b5686535c2d19fd3c24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf756d2a72414257ae7197d5ca26d0d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 } } peers { permanent_uuid: "a310b4a5969a4b5686535c2d19fd3c24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46153 } } peers { permanent_uuid: "d73297192af149f3b757984ddd9afcb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33961 } }
I20260925 04:39:23.666385 8447 leader_election.cc:288] T a90b585dac3e4eeaa4184fb2311b995b P a310b4a5969a4b5686535c2d19fd3c24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf756d2a72414257ae7197d5ca26d0d7 (127.0.98.2:42389), d73297192af149f3b757984ddd9afcb0 (127.0.98.1:33961)
I20260925 04:39:23.670504 8305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:23.666688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33961 (local address 127.0.98.3:52899)
0925 04:39:23.666804 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:23.666820 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:23.666918 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:23.667154 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:23.667157 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:23.667178 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:39:23.667303 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:23.667310 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:23.668251 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:23.668254 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:23.669363 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:23.669374 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:23.670078 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:23.670081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:23.670208 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:23.670234 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 04:39:23.670304 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 04:39:23.670385 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":4}
I20260925 04:39:23.670604 8439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:23.666779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52899 (local address 127.0.98.1:33961)
0925 04:39:23.666915 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:23.666920 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:23.666937 (+ 17us) server_negotiation.cc:407] Connection header received
0925 04:39:23.666995 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:23.666998 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:23.667047 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:23.667123 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:39:23.667462 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:23.668130 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:23.669509 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:23.669967 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:23.670049 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:23.670105 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:23.670360 (+ 255us) server_negotiation.cc:299] Negotiation successful
0925 04:39:23.670446 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 04:39:23.671536 8449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:23.666689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42389 (local address 127.0.98.3:44369)
0925 04:39:23.667419 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:23.667432 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:23.667520 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:23.667858 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:23.667860 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:23.667917 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:39:23.668025 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:23.668030 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:23.668778 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:23.668782 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:23.670216 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:23.670225 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:23.671105 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:23.671109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:23.671203 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:23.671228 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 04:39:23.671307 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 04:39:23.671384 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"mutex_wait_us":389,"thread_start_us":128,"threads_started":1}
I20260925 04:39:23.671437 8086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a90b585dac3e4eeaa4184fb2311b995b" candidate_uuid: "a310b4a5969a4b5686535c2d19fd3c24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73297192af149f3b757984ddd9afcb0" is_pre_election: true
I20260925 04:39:23.671562 8435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:23.666849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44369 (local address 127.0.98.2:42389)
0925 04:39:23.666960 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:23.666963 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:23.667521 (+ 558us) server_negotiation.cc:407] Connection header received
0925 04:39:23.667584 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:23.667587 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:23.667663 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:23.667743 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:39:23.668144 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:23.668650 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:23.670359 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:23.670948 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:23.671092 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:23.671157 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:23.671338 (+ 181us) server_negotiation.cc:299] Negotiation successful
0925 04:39:23.671414 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 04:39:23.671835 8086 raft_consensus.cc:2454] T a90b585dac3e4eeaa4184fb2311b995b P d73297192af149f3b757984ddd9afcb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a310b4a5969a4b5686535c2d19fd3c24 in term 0.
I20260925 04:39:23.672153 8226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a90b585dac3e4eeaa4184fb2311b995b" candidate_uuid: "a310b4a5969a4b5686535c2d19fd3c24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf756d2a72414257ae7197d5ca26d0d7" is_pre_election: true
I20260925 04:39:23.672443 8226 raft_consensus.cc:2454] T a90b585dac3e4eeaa4184fb2311b995b P bf756d2a72414257ae7197d5ca26d0d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a310b4a5969a4b5686535c2d19fd3c24 in term 0.
I20260925 04:39:23.672490 8300 leader_election.cc:302] T a90b585dac3e4eeaa4184fb2311b995b P a310b4a5969a4b5686535c2d19fd3c24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a310b4a5969a4b5686535c2d19fd3c24, d73297192af149f3b757984ddd9afcb0; no voters:
I20260925 04:39:23.672794 8447 raft_consensus.cc:2781] T a90b585dac3e4eeaa4184fb2311b995b P a310b4a5969a4b5686535c2d19fd3c24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:39:23.672884 8447 raft_consensus.cc:482] T a90b585dac3e4eeaa4184fb2311b995b P a310b4a5969a4b5686535c2d19fd3c24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:39:23.672926 8447 raft_consensus.cc:3037] T a90b585dac3e4eeaa4184fb2311b995b P a310b4a5969a4b5686535c2d19fd3c24 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:39:23.674003 8447 raft_consensus.cc:504] T a90b585dac3e4eeaa4184fb2311b995b P a310b4a5969a4b5686535c2d19fd3c24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf756d2a72414257ae7197d5ca26d0d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 } } peers { permanent_uuid: "a310b4a5969a4b5686535c2d19fd3c24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46153 } } peers { permanent_uuid: "d73297192af149f3b757984ddd9afcb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33961 } }
I20260925 04:39:23.674358 8447 leader_election.cc:288] T a90b585dac3e4eeaa4184fb2311b995b P a310b4a5969a4b5686535c2d19fd3c24 [CANDIDATE]: Term 1 election: Requested vote from peers bf756d2a72414257ae7197d5ca26d0d7 (127.0.98.2:42389), d73297192af149f3b757984ddd9afcb0 (127.0.98.1:33961)
I20260925 04:39:23.674759 8226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a90b585dac3e4eeaa4184fb2311b995b" candidate_uuid: "a310b4a5969a4b5686535c2d19fd3c24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf756d2a72414257ae7197d5ca26d0d7"
I20260925 04:39:23.674902 8086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a90b585dac3e4eeaa4184fb2311b995b" candidate_uuid: "a310b4a5969a4b5686535c2d19fd3c24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73297192af149f3b757984ddd9afcb0"
I20260925 04:39:23.674903 8226 raft_consensus.cc:3037] T a90b585dac3e4eeaa4184fb2311b995b P bf756d2a72414257ae7197d5ca26d0d7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:39:23.675025 8086 raft_consensus.cc:3037] T a90b585dac3e4eeaa4184fb2311b995b P d73297192af149f3b757984ddd9afcb0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:39:23.676288 8086 raft_consensus.cc:2454] T a90b585dac3e4eeaa4184fb2311b995b P d73297192af149f3b757984ddd9afcb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a310b4a5969a4b5686535c2d19fd3c24 in term 1.
I20260925 04:39:23.676287 8226 raft_consensus.cc:2454] T a90b585dac3e4eeaa4184fb2311b995b P bf756d2a72414257ae7197d5ca26d0d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a310b4a5969a4b5686535c2d19fd3c24 in term 1.
I20260925 04:39:23.676632 8297 leader_election.cc:302] T a90b585dac3e4eeaa4184fb2311b995b P a310b4a5969a4b5686535c2d19fd3c24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a310b4a5969a4b5686535c2d19fd3c24, bf756d2a72414257ae7197d5ca26d0d7; no voters:
I20260925 04:39:23.676789 8447 raft_consensus.cc:2781] T a90b585dac3e4eeaa4184fb2311b995b P a310b4a5969a4b5686535c2d19fd3c24 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:39:23.677013 8447 raft_consensus.cc:686] T a90b585dac3e4eeaa4184fb2311b995b P a310b4a5969a4b5686535c2d19fd3c24 [term 1 LEADER]: Becoming Leader. State: Replica: a310b4a5969a4b5686535c2d19fd3c24, State: Running, Role: LEADER
I20260925 04:39:23.677384 8447 consensus_queue.cc:237] T a90b585dac3e4eeaa4184fb2311b995b P a310b4a5969a4b5686535c2d19fd3c24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf756d2a72414257ae7197d5ca26d0d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 } } peers { permanent_uuid: "a310b4a5969a4b5686535c2d19fd3c24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46153 } } peers { permanent_uuid: "d73297192af149f3b757984ddd9afcb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33961 } }
I20260925 04:39:23.680608 7927 catalog_manager.cc:5495] T a90b585dac3e4eeaa4184fb2311b995b P a310b4a5969a4b5686535c2d19fd3c24 reported cstate change: term changed from 0 to 1, leader changed from <none> to a310b4a5969a4b5686535c2d19fd3c24 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "a310b4a5969a4b5686535c2d19fd3c24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf756d2a72414257ae7197d5ca26d0d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a310b4a5969a4b5686535c2d19fd3c24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46153 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d73297192af149f3b757984ddd9afcb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33961 } health_report { overall_health: UNKNOWN } } }
I20260925 04:39:23.696030 8438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:23.692521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39076 (local address 127.0.98.3:46153)
0925 04:39:23.692647 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:23.692651 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:23.692663 (+ 12us) server_negotiation.cc:407] Connection header received
0925 04:39:23.692731 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:23.692733 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:23.692788 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:23.692878 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:39:23.693156 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:23.693669 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:23.694598 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:23.694834 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:23.694916 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:23.695156 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:39:23.695652 (+ 496us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:39:23.695743 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:23.695827 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 04:39:23.695884 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 04:39:23.702056 8086 raft_consensus.cc:1261] T a90b585dac3e4eeaa4184fb2311b995b P d73297192af149f3b757984ddd9afcb0 [term 1 FOLLOWER]: Refusing update from remote peer a310b4a5969a4b5686535c2d19fd3c24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:39:23.702315 8226 raft_consensus.cc:1261] T a90b585dac3e4eeaa4184fb2311b995b P bf756d2a72414257ae7197d5ca26d0d7 [term 1 FOLLOWER]: Refusing update from remote peer a310b4a5969a4b5686535c2d19fd3c24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:39:23.702764 8450 consensus_queue.cc:1035] T a90b585dac3e4eeaa4184fb2311b995b P a310b4a5969a4b5686535c2d19fd3c24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73297192af149f3b757984ddd9afcb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:39:23.702955 8447 consensus_queue.cc:1035] T a90b585dac3e4eeaa4184fb2311b995b P a310b4a5969a4b5686535c2d19fd3c24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf756d2a72414257ae7197d5ca26d0d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:39:23.712188 8454 mvcc.cc:205] Tried to move back new op lower bound from 7333114526516658176 to 7333114526425354240. Current Snapshot: MvccSnapshot[applied={T|T < 7333114526516658176}]
I20260925 04:39:23.720446 7927 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:39:23.720636 7927 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:39:23.723959 7927 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bf7d385c164047fd9a6fc095948982f6: Sending DeleteTablet for 3 replicas of tablet a90b585dac3e4eeaa4184fb2311b995b
I20260925 04:39:23.724891 8206 tablet_service.cc:1514] Processing DeleteTablet for tablet a90b585dac3e4eeaa4184fb2311b995b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:39:23 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49130
I20260925 04:39:23.724949 8066 tablet_service.cc:1514] Processing DeleteTablet for tablet a90b585dac3e4eeaa4184fb2311b995b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:39:23 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43978
I20260925 04:39:23.725565 8463 tablet_replica.cc:330] T a90b585dac3e4eeaa4184fb2311b995b P d73297192af149f3b757984ddd9afcb0: stopping tablet replica
I20260925 04:39:23.725477 7927 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56456:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 04:39:23.725658 8346 tablet_service.cc:1514] Processing DeleteTablet for tablet a90b585dac3e4eeaa4184fb2311b995b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:39:23 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39064
I20260925 04:39:23.725873 8462 tablet_replica.cc:330] T a90b585dac3e4eeaa4184fb2311b995b P bf756d2a72414257ae7197d5ca26d0d7: stopping tablet replica
W20260925 04:39:23.725951 7927 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:39:23.726347 8463 raft_consensus.cc:2229] T a90b585dac3e4eeaa4184fb2311b995b P d73297192af149f3b757984ddd9afcb0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:39:23.726372 8464 tablet_replica.cc:330] T a90b585dac3e4eeaa4184fb2311b995b P a310b4a5969a4b5686535c2d19fd3c24: stopping tablet replica
I20260925 04:39:23.726683 8464 raft_consensus.cc:2229] T a90b585dac3e4eeaa4184fb2311b995b P a310b4a5969a4b5686535c2d19fd3c24 [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:39:23.726733 8462 raft_consensus.cc:2229] T a90b585dac3e4eeaa4184fb2311b995b P bf756d2a72414257ae7197d5ca26d0d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:39:23.727138 8464 raft_consensus.cc:2258] T a90b585dac3e4eeaa4184fb2311b995b P a310b4a5969a4b5686535c2d19fd3c24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:39:23.727486 8463 raft_consensus.cc:2258] T a90b585dac3e4eeaa4184fb2311b995b P d73297192af149f3b757984ddd9afcb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:39:23.727181 8462 raft_consensus.cc:2258] T a90b585dac3e4eeaa4184fb2311b995b P bf756d2a72414257ae7197d5ca26d0d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:39:23.728816 8464 ts_tablet_manager.cc:1905] T a90b585dac3e4eeaa4184fb2311b995b P a310b4a5969a4b5686535c2d19fd3c24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:39:23.730103 8463 ts_tablet_manager.cc:1905] T a90b585dac3e4eeaa4184fb2311b995b P d73297192af149f3b757984ddd9afcb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:39:23.731070 8462 ts_tablet_manager.cc:1905] T a90b585dac3e4eeaa4184fb2311b995b P bf756d2a72414257ae7197d5ca26d0d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:39:23.731980 8346 tablet_service.cc:1467] Processing CreateTablet for tablet 162898bd51f2441da989db58cb34241b (DEFAULT_TABLE table=test-table [id=ecbd8adf34644a66a8c0eb4287795c7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:39:23.732015 8066 tablet_service.cc:1467] Processing CreateTablet for tablet 162898bd51f2441da989db58cb34241b (DEFAULT_TABLE table=test-table [id=ecbd8adf34644a66a8c0eb4287795c7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:39:23.732260 8346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 162898bd51f2441da989db58cb34241b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:39:23.732342 8066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 162898bd51f2441da989db58cb34241b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:39:23.734748 8442 tablet_bootstrap.cc:492] T 162898bd51f2441da989db58cb34241b P d73297192af149f3b757984ddd9afcb0: Bootstrap starting.
I20260925 04:39:23.734863 8440 tablet_bootstrap.cc:492] T 162898bd51f2441da989db58cb34241b P a310b4a5969a4b5686535c2d19fd3c24: Bootstrap starting.
I20260925 04:39:23.735183 8206 tablet_service.cc:1467] Processing CreateTablet for tablet 162898bd51f2441da989db58cb34241b (DEFAULT_TABLE table=test-table [id=ecbd8adf34644a66a8c0eb4287795c7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:39:23.735466 8206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 162898bd51f2441da989db58cb34241b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:39:23.735893 8440 tablet_bootstrap.cc:654] T 162898bd51f2441da989db58cb34241b P a310b4a5969a4b5686535c2d19fd3c24: Neither blocks nor log segments found. Creating new log.
I20260925 04:39:23.735895 8442 tablet_bootstrap.cc:654] T 162898bd51f2441da989db58cb34241b P d73297192af149f3b757984ddd9afcb0: Neither blocks nor log segments found. Creating new log.
I20260925 04:39:23.736750 8464 ts_tablet_manager.cc:1918] T a90b585dac3e4eeaa4184fb2311b995b P a310b4a5969a4b5686535c2d19fd3c24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:39:23.736867 8464 log.cc:1196] T a90b585dac3e4eeaa4184fb2311b995b P a310b4a5969a4b5686535c2d19fd3c24: Deleting WAL directory at /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/ts-2/wal/wals/a90b585dac3e4eeaa4184fb2311b995b
I20260925 04:39:23.737195 8464 ts_tablet_manager.cc:1939] T a90b585dac3e4eeaa4184fb2311b995b P a310b4a5969a4b5686535c2d19fd3c24: Deleting consensus metadata
I20260925 04:39:23.738005 8441 tablet_bootstrap.cc:492] T 162898bd51f2441da989db58cb34241b P bf756d2a72414257ae7197d5ca26d0d7: Bootstrap starting.
I20260925 04:39:23.739092 8441 tablet_bootstrap.cc:654] T 162898bd51f2441da989db58cb34241b P bf756d2a72414257ae7197d5ca26d0d7: Neither blocks nor log segments found. Creating new log.
I20260925 04:39:23.740372 7916 catalog_manager.cc:4841] TS a310b4a5969a4b5686535c2d19fd3c24 (127.0.98.3:46153): tablet a90b585dac3e4eeaa4184fb2311b995b (table test-table [id=93fab460a95746f38c44945e35accf0d]) successfully deleted
I20260925 04:39:23.741282 8440 tablet_bootstrap.cc:492] T 162898bd51f2441da989db58cb34241b P a310b4a5969a4b5686535c2d19fd3c24: No bootstrap required, opened a new log
I20260925 04:39:23.741389 8440 ts_tablet_manager.cc:1397] T 162898bd51f2441da989db58cb34241b P a310b4a5969a4b5686535c2d19fd3c24: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260925 04:39:23.741420 8441 tablet_bootstrap.cc:492] T 162898bd51f2441da989db58cb34241b P bf756d2a72414257ae7197d5ca26d0d7: No bootstrap required, opened a new log
I20260925 04:39:23.741519 8441 ts_tablet_manager.cc:1397] T 162898bd51f2441da989db58cb34241b P bf756d2a72414257ae7197d5ca26d0d7: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260925 04:39:23.741928 8440 raft_consensus.cc:348] T 162898bd51f2441da989db58cb34241b P a310b4a5969a4b5686535c2d19fd3c24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73297192af149f3b757984ddd9afcb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33961 } } peers { permanent_uuid: "bf756d2a72414257ae7197d5ca26d0d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 } } peers { permanent_uuid: "a310b4a5969a4b5686535c2d19fd3c24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46153 } }
I20260925 04:39:23.741990 8441 raft_consensus.cc:348] T 162898bd51f2441da989db58cb34241b P bf756d2a72414257ae7197d5ca26d0d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73297192af149f3b757984ddd9afcb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33961 } } peers { permanent_uuid: "bf756d2a72414257ae7197d5ca26d0d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 } } peers { permanent_uuid: "a310b4a5969a4b5686535c2d19fd3c24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46153 } }
I20260925 04:39:23.742077 8440 raft_consensus.cc:374] T 162898bd51f2441da989db58cb34241b P a310b4a5969a4b5686535c2d19fd3c24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:39:23.742106 8441 raft_consensus.cc:374] T 162898bd51f2441da989db58cb34241b P bf756d2a72414257ae7197d5ca26d0d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:39:23.742112 8440 raft_consensus.cc:729] T 162898bd51f2441da989db58cb34241b P a310b4a5969a4b5686535c2d19fd3c24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a310b4a5969a4b5686535c2d19fd3c24, State: Initialized, Role: FOLLOWER
I20260925 04:39:23.742138 8441 raft_consensus.cc:729] T 162898bd51f2441da989db58cb34241b P bf756d2a72414257ae7197d5ca26d0d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf756d2a72414257ae7197d5ca26d0d7, State: Initialized, Role: FOLLOWER
I20260925 04:39:23.742244 8440 consensus_queue.cc:260] T 162898bd51f2441da989db58cb34241b P a310b4a5969a4b5686535c2d19fd3c24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73297192af149f3b757984ddd9afcb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33961 } } peers { permanent_uuid: "bf756d2a72414257ae7197d5ca26d0d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 } } peers { permanent_uuid: "a310b4a5969a4b5686535c2d19fd3c24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46153 } }
I20260925 04:39:23.742259 8441 consensus_queue.cc:260] T 162898bd51f2441da989db58cb34241b P bf756d2a72414257ae7197d5ca26d0d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73297192af149f3b757984ddd9afcb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33961 } } peers { permanent_uuid: "bf756d2a72414257ae7197d5ca26d0d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 } } peers { permanent_uuid: "a310b4a5969a4b5686535c2d19fd3c24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46153 } }
I20260925 04:39:23.742496 8440 ts_tablet_manager.cc:1428] T 162898bd51f2441da989db58cb34241b P a310b4a5969a4b5686535c2d19fd3c24: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 04:39:23.742507 8441 ts_tablet_manager.cc:1428] T 162898bd51f2441da989db58cb34241b P bf756d2a72414257ae7197d5ca26d0d7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 04:39:23.744534 8463 ts_tablet_manager.cc:1918] T a90b585dac3e4eeaa4184fb2311b995b P d73297192af149f3b757984ddd9afcb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:39:23.744661 8463 log.cc:1196] T a90b585dac3e4eeaa4184fb2311b995b P d73297192af149f3b757984ddd9afcb0: Deleting WAL directory at /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/ts-0/wal/wals/a90b585dac3e4eeaa4184fb2311b995b
I20260925 04:39:23.744976 8463 ts_tablet_manager.cc:1939] T a90b585dac3e4eeaa4184fb2311b995b P d73297192af149f3b757984ddd9afcb0: Deleting consensus metadata
I20260925 04:39:23.745353 8462 ts_tablet_manager.cc:1918] T a90b585dac3e4eeaa4184fb2311b995b P bf756d2a72414257ae7197d5ca26d0d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:39:23.745584 8462 log.cc:1196] T a90b585dac3e4eeaa4184fb2311b995b P bf756d2a72414257ae7197d5ca26d0d7: Deleting WAL directory at /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/ts-1/wal/wals/a90b585dac3e4eeaa4184fb2311b995b
I20260925 04:39:23.745886 8462 ts_tablet_manager.cc:1939] T a90b585dac3e4eeaa4184fb2311b995b P bf756d2a72414257ae7197d5ca26d0d7: Deleting consensus metadata
I20260925 04:39:23.746366 7916 catalog_manager.cc:4841] TS d73297192af149f3b757984ddd9afcb0 (127.0.98.1:33961): tablet a90b585dac3e4eeaa4184fb2311b995b (table test-table [id=93fab460a95746f38c44945e35accf0d]) successfully deleted
I20260925 04:39:23.747359 7913 catalog_manager.cc:4841] TS bf756d2a72414257ae7197d5ca26d0d7 (127.0.98.2:42389): tablet a90b585dac3e4eeaa4184fb2311b995b (table test-table [id=93fab460a95746f38c44945e35accf0d]) successfully deleted
I20260925 04:39:23.747520 8442 tablet_bootstrap.cc:492] T 162898bd51f2441da989db58cb34241b P d73297192af149f3b757984ddd9afcb0: No bootstrap required, opened a new log
I20260925 04:39:23.747641 8442 ts_tablet_manager.cc:1397] T 162898bd51f2441da989db58cb34241b P d73297192af149f3b757984ddd9afcb0: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20260925 04:39:23.748114 8442 raft_consensus.cc:348] T 162898bd51f2441da989db58cb34241b P d73297192af149f3b757984ddd9afcb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73297192af149f3b757984ddd9afcb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33961 } } peers { permanent_uuid: "bf756d2a72414257ae7197d5ca26d0d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 } } peers { permanent_uuid: "a310b4a5969a4b5686535c2d19fd3c24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46153 } }
I20260925 04:39:23.748241 8442 raft_consensus.cc:374] T 162898bd51f2441da989db58cb34241b P d73297192af149f3b757984ddd9afcb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:39:23.748272 8442 raft_consensus.cc:729] T 162898bd51f2441da989db58cb34241b P d73297192af149f3b757984ddd9afcb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73297192af149f3b757984ddd9afcb0, State: Initialized, Role: FOLLOWER
I20260925 04:39:23.748400 8442 consensus_queue.cc:260] T 162898bd51f2441da989db58cb34241b P d73297192af149f3b757984ddd9afcb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73297192af149f3b757984ddd9afcb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33961 } } peers { permanent_uuid: "bf756d2a72414257ae7197d5ca26d0d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 } } peers { permanent_uuid: "a310b4a5969a4b5686535c2d19fd3c24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46153 } }
I20260925 04:39:23.748672 8442 ts_tablet_manager.cc:1428] T 162898bd51f2441da989db58cb34241b P d73297192af149f3b757984ddd9afcb0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20260925 04:39:23.794422 8413 tablet.cc:2369] T 162898bd51f2441da989db58cb34241b P a310b4a5969a4b5686535c2d19fd3c24: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 04:39:23.825320 8273 tablet.cc:2369] T 162898bd51f2441da989db58cb34241b P bf756d2a72414257ae7197d5ca26d0d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 04:39:23.844249 8133 tablet.cc:2369] T 162898bd51f2441da989db58cb34241b P d73297192af149f3b757984ddd9afcb0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:39:23.950067 8446 raft_consensus.cc:482] T 162898bd51f2441da989db58cb34241b P bf756d2a72414257ae7197d5ca26d0d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:39:23.950233 8446 raft_consensus.cc:504] T 162898bd51f2441da989db58cb34241b P bf756d2a72414257ae7197d5ca26d0d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73297192af149f3b757984ddd9afcb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33961 } } peers { permanent_uuid: "bf756d2a72414257ae7197d5ca26d0d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 } } peers { permanent_uuid: "a310b4a5969a4b5686535c2d19fd3c24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46153 } }
I20260925 04:39:23.951267 8446 leader_election.cc:288] T 162898bd51f2441da989db58cb34241b P bf756d2a72414257ae7197d5ca26d0d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d73297192af149f3b757984ddd9afcb0 (127.0.98.1:33961), a310b4a5969a4b5686535c2d19fd3c24 (127.0.98.3:46153)
I20260925 04:39:23.955147 8439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:23.951694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33843 (local address 127.0.98.1:33961)
0925 04:39:23.951818 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:23.951822 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:23.951990 (+ 168us) server_negotiation.cc:407] Connection header received
0925 04:39:23.952054 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:23.952056 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:23.952108 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:23.952195 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:39:23.952517 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:23.953046 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:23.954215 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:23.954612 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:23.954703 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:23.954766 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:23.954939 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 04:39:23.955011 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 04:39:23.955173 8468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:23.951538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33961 (local address 127.0.98.2:33843)
0925 04:39:23.951827 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:23.951842 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:23.951954 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:23.952241 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:23.952245 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:23.952265 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:39:23.952381 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:23.952386 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:23.953152 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:23.953156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:23.954087 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:23.954097 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:23.954723 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:23.954727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:23.954825 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:23.954847 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 04:39:23.954911 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 04:39:23.955002 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":22,"spinlock_wait_cycles":2944,"thread_start_us":110,"threads_started":1}
I20260925 04:39:23.955219 8438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:23.952052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40831 (local address 127.0.98.3:46153)
0925 04:39:23.952172 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:23.952176 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:23.952268 (+ 92us) server_negotiation.cc:407] Connection header received
0925 04:39:23.952385 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:23.952388 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:23.952427 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:23.952486 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:39:23.952746 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:23.953313 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:23.954328 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:23.954725 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:23.954815 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:23.954875 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:23.955034 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 04:39:23.955116 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 04:39:23.955421 8469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:23.951940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46153 (local address 127.0.98.2:40831)
0925 04:39:23.952181 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:23.952193 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:23.952288 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:23.952524 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:23.952527 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:23.952550 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:39:23.952639 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:23.952649 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:23.953412 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:23.953415 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:23.954213 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:23.954221 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:23.954829 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:23.954833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:23.954923 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:23.954943 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 04:39:23.955010 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 04:39:23.955085 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":19,"spinlock_wait_cycles":3968,"thread_start_us":61,"threads_started":1}
I20260925 04:39:23.955790 8086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "162898bd51f2441da989db58cb34241b" candidate_uuid: "bf756d2a72414257ae7197d5ca26d0d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73297192af149f3b757984ddd9afcb0" is_pre_election: true
I20260925 04:39:23.955943 8086 raft_consensus.cc:2454] T 162898bd51f2441da989db58cb34241b P d73297192af149f3b757984ddd9afcb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf756d2a72414257ae7197d5ca26d0d7 in term 0.
I20260925 04:39:23.955972 8366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "162898bd51f2441da989db58cb34241b" candidate_uuid: "bf756d2a72414257ae7197d5ca26d0d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a310b4a5969a4b5686535c2d19fd3c24" is_pre_election: true
I20260925 04:39:23.956231 8366 raft_consensus.cc:2454] T 162898bd51f2441da989db58cb34241b P a310b4a5969a4b5686535c2d19fd3c24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf756d2a72414257ae7197d5ca26d0d7 in term 0.
I20260925 04:39:23.956298 8160 leader_election.cc:302] T 162898bd51f2441da989db58cb34241b P bf756d2a72414257ae7197d5ca26d0d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf756d2a72414257ae7197d5ca26d0d7, d73297192af149f3b757984ddd9afcb0; no voters:
I20260925 04:39:23.956557 8446 raft_consensus.cc:2781] T 162898bd51f2441da989db58cb34241b P bf756d2a72414257ae7197d5ca26d0d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:39:23.956616 8446 raft_consensus.cc:482] T 162898bd51f2441da989db58cb34241b P bf756d2a72414257ae7197d5ca26d0d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:39:23.956660 8446 raft_consensus.cc:3037] T 162898bd51f2441da989db58cb34241b P bf756d2a72414257ae7197d5ca26d0d7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:39:23.957602 8446 raft_consensus.cc:504] T 162898bd51f2441da989db58cb34241b P bf756d2a72414257ae7197d5ca26d0d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73297192af149f3b757984ddd9afcb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33961 } } peers { permanent_uuid: "bf756d2a72414257ae7197d5ca26d0d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 } } peers { permanent_uuid: "a310b4a5969a4b5686535c2d19fd3c24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46153 } }
I20260925 04:39:23.957916 8446 leader_election.cc:288] T 162898bd51f2441da989db58cb34241b P bf756d2a72414257ae7197d5ca26d0d7 [CANDIDATE]: Term 1 election: Requested vote from peers d73297192af149f3b757984ddd9afcb0 (127.0.98.1:33961), a310b4a5969a4b5686535c2d19fd3c24 (127.0.98.3:46153)
I20260925 04:39:23.958247 8086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "162898bd51f2441da989db58cb34241b" candidate_uuid: "bf756d2a72414257ae7197d5ca26d0d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73297192af149f3b757984ddd9afcb0"
I20260925 04:39:23.958309 8366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "162898bd51f2441da989db58cb34241b" candidate_uuid: "bf756d2a72414257ae7197d5ca26d0d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a310b4a5969a4b5686535c2d19fd3c24"
I20260925 04:39:23.958359 8086 raft_consensus.cc:3037] T 162898bd51f2441da989db58cb34241b P d73297192af149f3b757984ddd9afcb0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:39:23.958417 8366 raft_consensus.cc:3037] T 162898bd51f2441da989db58cb34241b P a310b4a5969a4b5686535c2d19fd3c24 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:39:23.959275 8366 raft_consensus.cc:2454] T 162898bd51f2441da989db58cb34241b P a310b4a5969a4b5686535c2d19fd3c24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf756d2a72414257ae7197d5ca26d0d7 in term 1.
I20260925 04:39:23.959275 8086 raft_consensus.cc:2454] T 162898bd51f2441da989db58cb34241b P d73297192af149f3b757984ddd9afcb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf756d2a72414257ae7197d5ca26d0d7 in term 1.
I20260925 04:39:23.959524 8160 leader_election.cc:302] T 162898bd51f2441da989db58cb34241b P bf756d2a72414257ae7197d5ca26d0d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a310b4a5969a4b5686535c2d19fd3c24, bf756d2a72414257ae7197d5ca26d0d7; no voters:
I20260925 04:39:23.959714 8446 raft_consensus.cc:2781] T 162898bd51f2441da989db58cb34241b P bf756d2a72414257ae7197d5ca26d0d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:39:23.959901 8446 raft_consensus.cc:686] T 162898bd51f2441da989db58cb34241b P bf756d2a72414257ae7197d5ca26d0d7 [term 1 LEADER]: Becoming Leader. State: Replica: bf756d2a72414257ae7197d5ca26d0d7, State: Running, Role: LEADER
I20260925 04:39:23.960152 8446 consensus_queue.cc:237] T 162898bd51f2441da989db58cb34241b P bf756d2a72414257ae7197d5ca26d0d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73297192af149f3b757984ddd9afcb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33961 } } peers { permanent_uuid: "bf756d2a72414257ae7197d5ca26d0d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 } } peers { permanent_uuid: "a310b4a5969a4b5686535c2d19fd3c24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46153 } }
I20260925 04:39:23.962261 7928 catalog_manager.cc:5495] T 162898bd51f2441da989db58cb34241b P bf756d2a72414257ae7197d5ca26d0d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf756d2a72414257ae7197d5ca26d0d7 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "bf756d2a72414257ae7197d5ca26d0d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73297192af149f3b757984ddd9afcb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf756d2a72414257ae7197d5ca26d0d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a310b4a5969a4b5686535c2d19fd3c24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46153 } health_report { overall_health: UNKNOWN } } }
Sep 25 04:39:23 dist-test-slave-dc95 krb5kdc[7884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311162, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 04:39:24.006292 7978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:23.997619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56464 (local address 127.0.98.62:40557)
0925 04:39:23.997784 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:23.997788 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:23.997897 (+ 109us) server_negotiation.cc:407] Connection header received
0925 04:39:23.998047 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:23.998050 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:23.998094 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:23.998175 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:23.998985 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:23.999517 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:24.000481 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:24.000708 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:24.000788 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:24.003219 (+ 2431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:24.003223 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:24.003224 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:24.003248 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:24.004708 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:24.005216 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:24.005219 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:24.005220 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:24.005292 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:24.005492 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:24.005494 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:24.005495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:24.005666 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:24.005734 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:24.005985 (+ 251us) server_negotiation.cc:299] Negotiation successful
0925 04:39:24.006163 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":75,"server-negotiator.queue_time_us":82}
I20260925 04:39:24.006294 8477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:23.997498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40557 (local address 127.0.0.1:56464)
0925 04:39:23.997817 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:23.997830 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:23.997933 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:23.998242 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:23.998245 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:23.998751 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:23.998861 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:23.998886 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:23.999684 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:23.999687 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:24.000361 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:24.000371 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:24.000801 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:24.000803 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:24.000871 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:24.001495 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:24.001506 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:24.003061 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:39:24.004824 (+ 1763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:24.004834 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:24.004840 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:24.005129 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:24.005360 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:24.005362 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:24.005363 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:24.005432 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:24.005737 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:39:24.005742 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:39:24.005852 (+ 110us) client_negotiation.cc:769] Sending connection context
0925 04:39:24.005937 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 04:39:24.006108 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"mutex_wait_us":1,"thread_start_us":126,"threads_started":1}
I20260925 04:39:24.010267 7928 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56464:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 04:39:24.011036 7928 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:39:24.016009 8066 tablet_service.cc:1467] Processing CreateTablet for tablet 4c52c289e58a42608287abbf65560116 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d1cbe1e8be7e4e93a9292457da597398]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 04:39:24.016207 8206 tablet_service.cc:1467] Processing CreateTablet for tablet 4c52c289e58a42608287abbf65560116 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d1cbe1e8be7e4e93a9292457da597398]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 04:39:24.016178 8346 tablet_service.cc:1467] Processing CreateTablet for tablet 4c52c289e58a42608287abbf65560116 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d1cbe1e8be7e4e93a9292457da597398]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 04:39:24.016518 8206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c52c289e58a42608287abbf65560116. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:39:24.016520 8066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c52c289e58a42608287abbf65560116. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:39:24.016566 8346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c52c289e58a42608287abbf65560116. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:39:24.019345 8440 tablet_bootstrap.cc:492] T 4c52c289e58a42608287abbf65560116 P a310b4a5969a4b5686535c2d19fd3c24: Bootstrap starting.
I20260925 04:39:24.019373 8442 tablet_bootstrap.cc:492] T 4c52c289e58a42608287abbf65560116 P d73297192af149f3b757984ddd9afcb0: Bootstrap starting.
I20260925 04:39:24.019567 8441 tablet_bootstrap.cc:492] T 4c52c289e58a42608287abbf65560116 P bf756d2a72414257ae7197d5ca26d0d7: Bootstrap starting.
I20260925 04:39:24.020212 8440 tablet_bootstrap.cc:654] T 4c52c289e58a42608287abbf65560116 P a310b4a5969a4b5686535c2d19fd3c24: Neither blocks nor log segments found. Creating new log.
I20260925 04:39:24.020391 8442 tablet_bootstrap.cc:654] T 4c52c289e58a42608287abbf65560116 P d73297192af149f3b757984ddd9afcb0: Neither blocks nor log segments found. Creating new log.
I20260925 04:39:24.020437 8441 tablet_bootstrap.cc:654] T 4c52c289e58a42608287abbf65560116 P bf756d2a72414257ae7197d5ca26d0d7: Neither blocks nor log segments found. Creating new log.
I20260925 04:39:24.021612 8441 tablet_bootstrap.cc:492] T 4c52c289e58a42608287abbf65560116 P bf756d2a72414257ae7197d5ca26d0d7: No bootstrap required, opened a new log
I20260925 04:39:24.021699 8441 ts_tablet_manager.cc:1397] T 4c52c289e58a42608287abbf65560116 P bf756d2a72414257ae7197d5ca26d0d7: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 04:39:24.021839 8442 tablet_bootstrap.cc:492] T 4c52c289e58a42608287abbf65560116 P d73297192af149f3b757984ddd9afcb0: No bootstrap required, opened a new log
I20260925 04:39:24.021906 8442 ts_tablet_manager.cc:1397] T 4c52c289e58a42608287abbf65560116 P d73297192af149f3b757984ddd9afcb0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 04:39:24.021970 8440 tablet_bootstrap.cc:492] T 4c52c289e58a42608287abbf65560116 P a310b4a5969a4b5686535c2d19fd3c24: No bootstrap required, opened a new log
I20260925 04:39:24.022045 8440 ts_tablet_manager.cc:1397] T 4c52c289e58a42608287abbf65560116 P a310b4a5969a4b5686535c2d19fd3c24: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 04:39:24.022161 8441 raft_consensus.cc:348] T 4c52c289e58a42608287abbf65560116 P bf756d2a72414257ae7197d5ca26d0d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73297192af149f3b757984ddd9afcb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33961 } } peers { permanent_uuid: "bf756d2a72414257ae7197d5ca26d0d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 } } peers { permanent_uuid: "a310b4a5969a4b5686535c2d19fd3c24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46153 } }
I20260925 04:39:24.022272 8441 raft_consensus.cc:374] T 4c52c289e58a42608287abbf65560116 P bf756d2a72414257ae7197d5ca26d0d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:39:24.022306 8441 raft_consensus.cc:729] T 4c52c289e58a42608287abbf65560116 P bf756d2a72414257ae7197d5ca26d0d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf756d2a72414257ae7197d5ca26d0d7, State: Initialized, Role: FOLLOWER
I20260925 04:39:24.022280 8442 raft_consensus.cc:348] T 4c52c289e58a42608287abbf65560116 P d73297192af149f3b757984ddd9afcb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73297192af149f3b757984ddd9afcb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33961 } } peers { permanent_uuid: "bf756d2a72414257ae7197d5ca26d0d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 } } peers { permanent_uuid: "a310b4a5969a4b5686535c2d19fd3c24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46153 } }
I20260925 04:39:24.022374 8442 raft_consensus.cc:374] T 4c52c289e58a42608287abbf65560116 P d73297192af149f3b757984ddd9afcb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:39:24.022404 8442 raft_consensus.cc:729] T 4c52c289e58a42608287abbf65560116 P d73297192af149f3b757984ddd9afcb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73297192af149f3b757984ddd9afcb0, State: Initialized, Role: FOLLOWER
I20260925 04:39:24.022351 8440 raft_consensus.cc:348] T 4c52c289e58a42608287abbf65560116 P a310b4a5969a4b5686535c2d19fd3c24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73297192af149f3b757984ddd9afcb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33961 } } peers { permanent_uuid: "bf756d2a72414257ae7197d5ca26d0d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 } } peers { permanent_uuid: "a310b4a5969a4b5686535c2d19fd3c24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46153 } }
I20260925 04:39:24.022445 8440 raft_consensus.cc:374] T 4c52c289e58a42608287abbf65560116 P a310b4a5969a4b5686535c2d19fd3c24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:39:24.022481 8440 raft_consensus.cc:729] T 4c52c289e58a42608287abbf65560116 P a310b4a5969a4b5686535c2d19fd3c24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a310b4a5969a4b5686535c2d19fd3c24, State: Initialized, Role: FOLLOWER
I20260925 04:39:24.022447 8441 consensus_queue.cc:260] T 4c52c289e58a42608287abbf65560116 P bf756d2a72414257ae7197d5ca26d0d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73297192af149f3b757984ddd9afcb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33961 } } peers { permanent_uuid: "bf756d2a72414257ae7197d5ca26d0d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 } } peers { permanent_uuid: "a310b4a5969a4b5686535c2d19fd3c24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46153 } }
I20260925 04:39:24.022527 8442 consensus_queue.cc:260] T 4c52c289e58a42608287abbf65560116 P d73297192af149f3b757984ddd9afcb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73297192af149f3b757984ddd9afcb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33961 } } peers { permanent_uuid: "bf756d2a72414257ae7197d5ca26d0d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 } } peers { permanent_uuid: "a310b4a5969a4b5686535c2d19fd3c24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46153 } }
I20260925 04:39:24.022714 8441 ts_tablet_manager.cc:1428] T 4c52c289e58a42608287abbf65560116 P bf756d2a72414257ae7197d5ca26d0d7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 04:39:24.022758 8442 ts_tablet_manager.cc:1428] T 4c52c289e58a42608287abbf65560116 P d73297192af149f3b757984ddd9afcb0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 04:39:24.022680 8440 consensus_queue.cc:260] T 4c52c289e58a42608287abbf65560116 P a310b4a5969a4b5686535c2d19fd3c24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73297192af149f3b757984ddd9afcb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33961 } } peers { permanent_uuid: "bf756d2a72414257ae7197d5ca26d0d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 } } peers { permanent_uuid: "a310b4a5969a4b5686535c2d19fd3c24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46153 } }
I20260925 04:39:24.022780 8446 tablet_replica.cc:439] T 4c52c289e58a42608287abbf65560116 P bf756d2a72414257ae7197d5ca26d0d7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73297192af149f3b757984ddd9afcb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33961 } } peers { permanent_uuid: "bf756d2a72414257ae7197d5ca26d0d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 } } peers { permanent_uuid: "a310b4a5969a4b5686535c2d19fd3c24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46153 } } }
I20260925 04:39:24.022931 8446 tablet_replica.cc:442] T 4c52c289e58a42608287abbf65560116 P bf756d2a72414257ae7197d5ca26d0d7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:39:24.023052 8440 ts_tablet_manager.cc:1428] T 4c52c289e58a42608287abbf65560116 P a310b4a5969a4b5686535c2d19fd3c24: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 04:39:24.023278 8448 tablet_replica.cc:439] T 4c52c289e58a42608287abbf65560116 P d73297192af149f3b757984ddd9afcb0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73297192af149f3b757984ddd9afcb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33961 } } peers { permanent_uuid: "bf756d2a72414257ae7197d5ca26d0d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 } } peers { permanent_uuid: "a310b4a5969a4b5686535c2d19fd3c24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46153 } } }
I20260925 04:39:24.023415 8448 tablet_replica.cc:442] T 4c52c289e58a42608287abbf65560116 P d73297192af149f3b757984ddd9afcb0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:39:24.023587 8450 tablet_replica.cc:439] T 4c52c289e58a42608287abbf65560116 P a310b4a5969a4b5686535c2d19fd3c24: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73297192af149f3b757984ddd9afcb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33961 } } peers { permanent_uuid: "bf756d2a72414257ae7197d5ca26d0d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 } } peers { permanent_uuid: "a310b4a5969a4b5686535c2d19fd3c24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46153 } } }
I20260925 04:39:24.023741 8450 tablet_replica.cc:442] T 4c52c289e58a42608287abbf65560116 P a310b4a5969a4b5686535c2d19fd3c24: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:39:24.053987 8450 raft_consensus.cc:482] T 4c52c289e58a42608287abbf65560116 P a310b4a5969a4b5686535c2d19fd3c24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:39:24.054145 8450 raft_consensus.cc:504] T 4c52c289e58a42608287abbf65560116 P a310b4a5969a4b5686535c2d19fd3c24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73297192af149f3b757984ddd9afcb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33961 } } peers { permanent_uuid: "bf756d2a72414257ae7197d5ca26d0d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 } } peers { permanent_uuid: "a310b4a5969a4b5686535c2d19fd3c24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46153 } }
I20260925 04:39:24.054538 8450 leader_election.cc:288] T 4c52c289e58a42608287abbf65560116 P a310b4a5969a4b5686535c2d19fd3c24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d73297192af149f3b757984ddd9afcb0 (127.0.98.1:33961), bf756d2a72414257ae7197d5ca26d0d7 (127.0.98.2:42389)
I20260925 04:39:24.054983 8086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c52c289e58a42608287abbf65560116" candidate_uuid: "a310b4a5969a4b5686535c2d19fd3c24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73297192af149f3b757984ddd9afcb0" is_pre_election: true
I20260925 04:39:24.055001 8226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c52c289e58a42608287abbf65560116" candidate_uuid: "a310b4a5969a4b5686535c2d19fd3c24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf756d2a72414257ae7197d5ca26d0d7" is_pre_election: true
I20260925 04:39:24.055141 8226 raft_consensus.cc:2454] T 4c52c289e58a42608287abbf65560116 P bf756d2a72414257ae7197d5ca26d0d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a310b4a5969a4b5686535c2d19fd3c24 in term 0.
I20260925 04:39:24.055140 8086 raft_consensus.cc:2454] T 4c52c289e58a42608287abbf65560116 P d73297192af149f3b757984ddd9afcb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a310b4a5969a4b5686535c2d19fd3c24 in term 0.
I20260925 04:39:24.055445 8300 leader_election.cc:302] T 4c52c289e58a42608287abbf65560116 P a310b4a5969a4b5686535c2d19fd3c24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a310b4a5969a4b5686535c2d19fd3c24, d73297192af149f3b757984ddd9afcb0; no voters:
I20260925 04:39:24.055651 8450 raft_consensus.cc:2781] T 4c52c289e58a42608287abbf65560116 P a310b4a5969a4b5686535c2d19fd3c24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:39:24.055722 8450 raft_consensus.cc:482] T 4c52c289e58a42608287abbf65560116 P a310b4a5969a4b5686535c2d19fd3c24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:39:24.055771 8450 raft_consensus.cc:3037] T 4c52c289e58a42608287abbf65560116 P a310b4a5969a4b5686535c2d19fd3c24 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:39:24.057044 8450 raft_consensus.cc:504] T 4c52c289e58a42608287abbf65560116 P a310b4a5969a4b5686535c2d19fd3c24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73297192af149f3b757984ddd9afcb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33961 } } peers { permanent_uuid: "bf756d2a72414257ae7197d5ca26d0d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 } } peers { permanent_uuid: "a310b4a5969a4b5686535c2d19fd3c24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46153 } }
I20260925 04:39:24.057405 8450 leader_election.cc:288] T 4c52c289e58a42608287abbf65560116 P a310b4a5969a4b5686535c2d19fd3c24 [CANDIDATE]: Term 1 election: Requested vote from peers d73297192af149f3b757984ddd9afcb0 (127.0.98.1:33961), bf756d2a72414257ae7197d5ca26d0d7 (127.0.98.2:42389)
I20260925 04:39:24.057804 8086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c52c289e58a42608287abbf65560116" candidate_uuid: "a310b4a5969a4b5686535c2d19fd3c24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73297192af149f3b757984ddd9afcb0"
I20260925 04:39:24.057847 8226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c52c289e58a42608287abbf65560116" candidate_uuid: "a310b4a5969a4b5686535c2d19fd3c24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf756d2a72414257ae7197d5ca26d0d7"
I20260925 04:39:24.057935 8086 raft_consensus.cc:3037] T 4c52c289e58a42608287abbf65560116 P d73297192af149f3b757984ddd9afcb0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:39:24.057961 8226 raft_consensus.cc:3037] T 4c52c289e58a42608287abbf65560116 P bf756d2a72414257ae7197d5ca26d0d7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:39:24.058849 8086 raft_consensus.cc:2454] T 4c52c289e58a42608287abbf65560116 P d73297192af149f3b757984ddd9afcb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a310b4a5969a4b5686535c2d19fd3c24 in term 1.
I20260925 04:39:24.058955 8226 raft_consensus.cc:2454] T 4c52c289e58a42608287abbf65560116 P bf756d2a72414257ae7197d5ca26d0d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a310b4a5969a4b5686535c2d19fd3c24 in term 1.
I20260925 04:39:24.059196 8300 leader_election.cc:302] T 4c52c289e58a42608287abbf65560116 P a310b4a5969a4b5686535c2d19fd3c24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a310b4a5969a4b5686535c2d19fd3c24, d73297192af149f3b757984ddd9afcb0; no voters:
I20260925 04:39:24.059394 8450 raft_consensus.cc:2781] T 4c52c289e58a42608287abbf65560116 P a310b4a5969a4b5686535c2d19fd3c24 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:39:24.059511 8450 raft_consensus.cc:686] T 4c52c289e58a42608287abbf65560116 P a310b4a5969a4b5686535c2d19fd3c24 [term 1 LEADER]: Becoming Leader. State: Replica: a310b4a5969a4b5686535c2d19fd3c24, State: Running, Role: LEADER
I20260925 04:39:24.059686 8450 consensus_queue.cc:237] T 4c52c289e58a42608287abbf65560116 P a310b4a5969a4b5686535c2d19fd3c24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73297192af149f3b757984ddd9afcb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33961 } } peers { permanent_uuid: "bf756d2a72414257ae7197d5ca26d0d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 } } peers { permanent_uuid: "a310b4a5969a4b5686535c2d19fd3c24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46153 } }
I20260925 04:39:24.060467 8447 tablet_replica.cc:439] T 4c52c289e58a42608287abbf65560116 P a310b4a5969a4b5686535c2d19fd3c24: TxnStatusTablet state changed. Reason: New leader a310b4a5969a4b5686535c2d19fd3c24. Latest consensus state: current_term: 1 leader_uuid: "a310b4a5969a4b5686535c2d19fd3c24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73297192af149f3b757984ddd9afcb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33961 } } peers { permanent_uuid: "bf756d2a72414257ae7197d5ca26d0d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 } } peers { permanent_uuid: "a310b4a5969a4b5686535c2d19fd3c24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46153 } } }
I20260925 04:39:24.060590 8447 tablet_replica.cc:442] T 4c52c289e58a42608287abbf65560116 P a310b4a5969a4b5686535c2d19fd3c24: This TxnStatusTablet replica's current role is: LEADER
I20260925 04:39:24.060820 8482 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 04:39:24.060976 8482 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 04:39:24.061695 7928 catalog_manager.cc:5495] T 4c52c289e58a42608287abbf65560116 P a310b4a5969a4b5686535c2d19fd3c24 reported cstate change: term changed from 0 to 1, leader changed from <none> to a310b4a5969a4b5686535c2d19fd3c24 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "a310b4a5969a4b5686535c2d19fd3c24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73297192af149f3b757984ddd9afcb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf756d2a72414257ae7197d5ca26d0d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a310b4a5969a4b5686535c2d19fd3c24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46153 } health_report { overall_health: HEALTHY } } }
I20260925 04:39:24.076066 8477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:24.072918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46153 (local address 127.0.0.1:39086)
0925 04:39:24.073067 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:24.073082 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:24.073183 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:24.073430 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:24.073433 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:24.073440 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 04:39:24.073536 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:24.073540 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:24.074256 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:24.074259 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:24.074873 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:24.074882 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:24.075298 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:24.075302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:24.075380 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:24.075428 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 04:39:24.075753 (+ 325us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 04:39:24.075770 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 04:39:24.075830 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 04:39:24.075927 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 04:39:24.076056 8438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:24.073096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39086 (local address 127.0.98.3:46153)
0925 04:39:24.073226 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:24.073231 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:24.073245 (+ 14us) server_negotiation.cc:407] Connection header received
0925 04:39:24.073293 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:24.073296 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:24.073341 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:24.073403 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:39:24.073629 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:24.074152 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:24.074984 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:24.075200 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:24.075286 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:24.075561 (+ 275us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:39:24.075678 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:39:24.075740 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:24.075850 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 04:39:24.075909 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 04:39:24.078652 8086 raft_consensus.cc:1261] T 4c52c289e58a42608287abbf65560116 P d73297192af149f3b757984ddd9afcb0 [term 1 FOLLOWER]: Refusing update from remote peer a310b4a5969a4b5686535c2d19fd3c24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:39:24.079066 8226 raft_consensus.cc:1261] T 4c52c289e58a42608287abbf65560116 P bf756d2a72414257ae7197d5ca26d0d7 [term 1 FOLLOWER]: Refusing update from remote peer a310b4a5969a4b5686535c2d19fd3c24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:39:24.079095 8447 consensus_queue.cc:1035] T 4c52c289e58a42608287abbf65560116 P a310b4a5969a4b5686535c2d19fd3c24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73297192af149f3b757984ddd9afcb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:39:24.079470 8450 consensus_queue.cc:1035] T 4c52c289e58a42608287abbf65560116 P a310b4a5969a4b5686535c2d19fd3c24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf756d2a72414257ae7197d5ca26d0d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:39:24.080860 8446 tablet_replica.cc:439] T 4c52c289e58a42608287abbf65560116 P bf756d2a72414257ae7197d5ca26d0d7: TxnStatusTablet state changed. Reason: New leader a310b4a5969a4b5686535c2d19fd3c24. Latest consensus state: current_term: 1 leader_uuid: "a310b4a5969a4b5686535c2d19fd3c24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73297192af149f3b757984ddd9afcb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33961 } } peers { permanent_uuid: "bf756d2a72414257ae7197d5ca26d0d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 } } peers { permanent_uuid: "a310b4a5969a4b5686535c2d19fd3c24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46153 } } }
I20260925 04:39:24.080997 8446 tablet_replica.cc:442] T 4c52c289e58a42608287abbf65560116 P bf756d2a72414257ae7197d5ca26d0d7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:39:24.080935 8448 tablet_replica.cc:439] T 4c52c289e58a42608287abbf65560116 P d73297192af149f3b757984ddd9afcb0: TxnStatusTablet state changed. Reason: New leader a310b4a5969a4b5686535c2d19fd3c24. Latest consensus state: current_term: 1 leader_uuid: "a310b4a5969a4b5686535c2d19fd3c24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73297192af149f3b757984ddd9afcb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33961 } } peers { permanent_uuid: "bf756d2a72414257ae7197d5ca26d0d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 } } peers { permanent_uuid: "a310b4a5969a4b5686535c2d19fd3c24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46153 } } }
I20260925 04:39:24.081053 8448 tablet_replica.cc:442] T 4c52c289e58a42608287abbf65560116 P d73297192af149f3b757984ddd9afcb0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:39:24.082087 8447 tablet_replica.cc:439] T 4c52c289e58a42608287abbf65560116 P a310b4a5969a4b5686535c2d19fd3c24: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a310b4a5969a4b5686535c2d19fd3c24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73297192af149f3b757984ddd9afcb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33961 } } peers { permanent_uuid: "bf756d2a72414257ae7197d5ca26d0d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 } } peers { permanent_uuid: "a310b4a5969a4b5686535c2d19fd3c24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46153 } } }
I20260925 04:39:24.082195 8447 tablet_replica.cc:442] T 4c52c289e58a42608287abbf65560116 P a310b4a5969a4b5686535c2d19fd3c24: This TxnStatusTablet replica's current role is: LEADER
I20260925 04:39:24.082609 8453 mvcc.cc:205] Tried to move back new op lower bound from 7333114528062226432 to 7333114527990251520. Current Snapshot: MvccSnapshot[applied={T|T < 7333114528062226432 or (T in {7333114528062226432})}]
I20260925 04:39:24.083073 8455 mvcc.cc:205] Tried to move back new op lower bound from 7333114528062226432 to 7333114527990251520. Current Snapshot: MvccSnapshot[applied={T|T < 7333114528062226432}]
I20260925 04:39:24.083261 8448 tablet_replica.cc:439] T 4c52c289e58a42608287abbf65560116 P d73297192af149f3b757984ddd9afcb0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a310b4a5969a4b5686535c2d19fd3c24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73297192af149f3b757984ddd9afcb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33961 } } peers { permanent_uuid: "bf756d2a72414257ae7197d5ca26d0d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 } } peers { permanent_uuid: "a310b4a5969a4b5686535c2d19fd3c24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46153 } } }
I20260925 04:39:24.083343 8448 tablet_replica.cc:442] T 4c52c289e58a42608287abbf65560116 P d73297192af149f3b757984ddd9afcb0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:39:24.087883 8450 tablet_replica.cc:439] T 4c52c289e58a42608287abbf65560116 P a310b4a5969a4b5686535c2d19fd3c24: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a310b4a5969a4b5686535c2d19fd3c24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73297192af149f3b757984ddd9afcb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33961 } } peers { permanent_uuid: "bf756d2a72414257ae7197d5ca26d0d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 } } peers { permanent_uuid: "a310b4a5969a4b5686535c2d19fd3c24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46153 } } }
I20260925 04:39:24.088023 8450 tablet_replica.cc:442] T 4c52c289e58a42608287abbf65560116 P a310b4a5969a4b5686535c2d19fd3c24: This TxnStatusTablet replica's current role is: LEADER
I20260925 04:39:24.089325 8446 tablet_replica.cc:439] T 4c52c289e58a42608287abbf65560116 P bf756d2a72414257ae7197d5ca26d0d7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a310b4a5969a4b5686535c2d19fd3c24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73297192af149f3b757984ddd9afcb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33961 } } peers { permanent_uuid: "bf756d2a72414257ae7197d5ca26d0d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 } } peers { permanent_uuid: "a310b4a5969a4b5686535c2d19fd3c24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46153 } } }
I20260925 04:39:24.089458 8446 tablet_replica.cc:442] T 4c52c289e58a42608287abbf65560116 P bf756d2a72414257ae7197d5ca26d0d7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:39:24.092625 8435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:24.087445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49136 (local address 127.0.98.2:42389)
0925 04:39:24.087550 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:24.087554 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:24.087656 (+ 102us) server_negotiation.cc:407] Connection header received
0925 04:39:24.087864 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:24.087868 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:24.087946 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:24.088043 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:39:24.088379 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:24.089181 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:24.090448 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:24.090796 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:24.090889 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:24.091197 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:39:24.091805 (+ 608us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:39:24.091911 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:24.092342 (+ 431us) server_negotiation.cc:299] Negotiation successful
0925 04:39:24.092408 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 04:39:24.108215 8438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:24.104220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59741 (local address 127.0.98.3:46153)
0925 04:39:24.104325 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:24.104330 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:24.104347 (+ 17us) server_negotiation.cc:407] Connection header received
0925 04:39:24.104402 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:24.104404 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:24.104446 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:24.104508 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:39:24.104801 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:24.105616 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:24.106906 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:24.107257 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:24.107359 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:24.107728 (+ 369us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:39:24.107822 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:39:24.107891 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:24.108001 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 04:39:24.108069 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 04:39:24.108610 8490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:24.103824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46153 (local address 127.0.98.2:59741)
0925 04:39:24.104183 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:24.104199 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:24.104305 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:24.104538 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:24.104540 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:24.104548 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 04:39:24.104673 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:24.104679 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:24.105743 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:24.105747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:24.106740 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:24.106754 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:24.107402 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:24.107406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:24.107505 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:24.107567 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 04:39:24.107910 (+ 343us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 04:39:24.107916 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 04:39:24.107983 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 04:39:24.108084 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":123,"threads_started":1}
I20260925 04:39:24.116456 8086 raft_consensus.cc:1261] T 162898bd51f2441da989db58cb34241b P d73297192af149f3b757984ddd9afcb0 [term 1 FOLLOWER]: Refusing update from remote peer bf756d2a72414257ae7197d5ca26d0d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:39:24.116595 8366 raft_consensus.cc:1261] T 162898bd51f2441da989db58cb34241b P a310b4a5969a4b5686535c2d19fd3c24 [term 1 FOLLOWER]: Refusing update from remote peer bf756d2a72414257ae7197d5ca26d0d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:39:24.117020 8470 consensus_queue.cc:1035] T 162898bd51f2441da989db58cb34241b P bf756d2a72414257ae7197d5ca26d0d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73297192af149f3b757984ddd9afcb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:39:24.117264 8446 consensus_queue.cc:1035] T 162898bd51f2441da989db58cb34241b P bf756d2a72414257ae7197d5ca26d0d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a310b4a5969a4b5686535c2d19fd3c24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:39:24.139528 8435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:24.136173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43577 (local address 127.0.98.2:42389)
0925 04:39:24.136289 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:24.136293 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:24.136309 (+ 16us) server_negotiation.cc:407] Connection header received
0925 04:39:24.136401 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:24.136404 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:24.136456 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:24.136530 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:39:24.136863 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:24.137496 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:24.138360 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:24.138595 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:24.138692 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:24.139051 (+ 359us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:39:24.139155 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:39:24.139220 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:24.139326 (+ 106us) server_negotiation.cc:299] Negotiation successful
0925 04:39:24.139379 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 04:39:24.139528 8495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:24.135879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42389 (local address 127.0.98.3:43577)
0925 04:39:24.136208 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:24.136223 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:24.136313 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:24.136661 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:24.136664 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:24.136671 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 04:39:24.136755 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:24.136760 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:24.137609 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:24.137612 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:24.138240 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:24.138249 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:24.138726 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:24.138731 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:24.138834 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:24.138897 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 04:39:24.139243 (+ 346us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 04:39:24.139250 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 04:39:24.139306 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 04:39:24.139395 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":101,"threads_started":1}
I20260925 04:39:24.168032 7928 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:39:24.168238 7928 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:39:24.170954 7928 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bf7d385c164047fd9a6fc095948982f6: Sending DeleteTablet for 3 replicas of tablet 162898bd51f2441da989db58cb34241b
I20260925 04:39:24.171926 8206 tablet_service.cc:1514] Processing DeleteTablet for tablet 162898bd51f2441da989db58cb34241b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:39:24 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49130
I20260925 04:39:24.171926 8066 tablet_service.cc:1514] Processing DeleteTablet for tablet 162898bd51f2441da989db58cb34241b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:39:24 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43978
I20260925 04:39:24.172147 8463 tablet_replica.cc:330] T 162898bd51f2441da989db58cb34241b P d73297192af149f3b757984ddd9afcb0: stopping tablet replica
I20260925 04:39:24.172195 8462 tablet_replica.cc:330] T 162898bd51f2441da989db58cb34241b P bf756d2a72414257ae7197d5ca26d0d7: stopping tablet replica
I20260925 04:39:24.172262 8463 raft_consensus.cc:2229] T 162898bd51f2441da989db58cb34241b P d73297192af149f3b757984ddd9afcb0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:39:24.172317 8462 raft_consensus.cc:2229] T 162898bd51f2441da989db58cb34241b P bf756d2a72414257ae7197d5ca26d0d7 [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:39:24.172386 8463 raft_consensus.cc:2258] T 162898bd51f2441da989db58cb34241b P d73297192af149f3b757984ddd9afcb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:39:24.172650 8462 raft_consensus.cc:2258] T 162898bd51f2441da989db58cb34241b P bf756d2a72414257ae7197d5ca26d0d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:39:24.173524 8463 ts_tablet_manager.cc:1905] T 162898bd51f2441da989db58cb34241b P d73297192af149f3b757984ddd9afcb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:39:24.175911 8346 tablet_service.cc:1514] Processing DeleteTablet for tablet 162898bd51f2441da989db58cb34241b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:39:24 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39064
I20260925 04:39:24.176132 8464 tablet_replica.cc:330] T 162898bd51f2441da989db58cb34241b P a310b4a5969a4b5686535c2d19fd3c24: stopping tablet replica
I20260925 04:39:24.176247 8464 raft_consensus.cc:2229] T 162898bd51f2441da989db58cb34241b P a310b4a5969a4b5686535c2d19fd3c24 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:39:24.176368 8464 raft_consensus.cc:2258] T 162898bd51f2441da989db58cb34241b P a310b4a5969a4b5686535c2d19fd3c24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:39:24.176755 8462 ts_tablet_manager.cc:1905] T 162898bd51f2441da989db58cb34241b P bf756d2a72414257ae7197d5ca26d0d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:39:24.178148 8463 ts_tablet_manager.cc:1918] T 162898bd51f2441da989db58cb34241b P d73297192af149f3b757984ddd9afcb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 04:39:24.178310 8463 log.cc:1196] T 162898bd51f2441da989db58cb34241b P d73297192af149f3b757984ddd9afcb0: Deleting WAL directory at /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/ts-0/wal/wals/162898bd51f2441da989db58cb34241b
I20260925 04:39:24.178740 8463 ts_tablet_manager.cc:1939] T 162898bd51f2441da989db58cb34241b P d73297192af149f3b757984ddd9afcb0: Deleting consensus metadata
I20260925 04:39:24.179738 7916 catalog_manager.cc:4841] TS d73297192af149f3b757984ddd9afcb0 (127.0.98.1:33961): tablet 162898bd51f2441da989db58cb34241b (table test-table [id=ecbd8adf34644a66a8c0eb4287795c7c]) successfully deleted
I20260925 04:39:24.180568 8462 ts_tablet_manager.cc:1918] T 162898bd51f2441da989db58cb34241b P bf756d2a72414257ae7197d5ca26d0d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 04:39:24.180660 8462 log.cc:1196] T 162898bd51f2441da989db58cb34241b P bf756d2a72414257ae7197d5ca26d0d7: Deleting WAL directory at /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/ts-1/wal/wals/162898bd51f2441da989db58cb34241b
I20260925 04:39:24.180902 8462 ts_tablet_manager.cc:1939] T 162898bd51f2441da989db58cb34241b P bf756d2a72414257ae7197d5ca26d0d7: Deleting consensus metadata
I20260925 04:39:24.181679 7913 catalog_manager.cc:4841] TS bf756d2a72414257ae7197d5ca26d0d7 (127.0.98.2:42389): tablet 162898bd51f2441da989db58cb34241b (table test-table [id=ecbd8adf34644a66a8c0eb4287795c7c]) successfully deleted
I20260925 04:39:24.182546 8464 ts_tablet_manager.cc:1905] T 162898bd51f2441da989db58cb34241b P a310b4a5969a4b5686535c2d19fd3c24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:39:24.185916 8464 ts_tablet_manager.cc:1918] T 162898bd51f2441da989db58cb34241b P a310b4a5969a4b5686535c2d19fd3c24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 04:39:24.186105 8464 log.cc:1196] T 162898bd51f2441da989db58cb34241b P a310b4a5969a4b5686535c2d19fd3c24: Deleting WAL directory at /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/ts-2/wal/wals/162898bd51f2441da989db58cb34241b
I20260925 04:39:24.186362 8464 ts_tablet_manager.cc:1939] T 162898bd51f2441da989db58cb34241b P a310b4a5969a4b5686535c2d19fd3c24: Deleting consensus metadata
I20260925 04:39:24.187213 7916 catalog_manager.cc:4841] TS a310b4a5969a4b5686535c2d19fd3c24 (127.0.98.3:46153): tablet 162898bd51f2441da989db58cb34241b (table test-table [id=ecbd8adf34644a66a8c0eb4287795c7c]) successfully deleted
Sep 25 04:39:24 dist-test-slave-dc95 krb5kdc[7884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 04:39:24 dist-test-slave-dc95 krb5kdc[7884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:39:24 dist-test-slave-dc95 krb5kdc[7884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 04:39:24.211962 7978 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:39:24.204057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56470 (local address 127.0.98.62:40557)
0925 04:39:24.204187 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:24.204192 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:24.204279 (+ 87us) server_negotiation.cc:407] Connection header received
0925 04:39:24.204417 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:24.204421 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:24.204473 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:24.204564 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:24.205393 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:24.206219 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:24.207564 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:24.207964 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:24.208083 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:24.211764 (+ 3681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56470: BlockingRecv error: recv got EOF from 127.0.0.1:56470 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 25 04:39:24 dist-test-slave-dc95 krb5kdc[7884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 04:39:24.224324 7978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:24.214101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56480 (local address 127.0.98.62:40557)
0925 04:39:24.214218 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:24.214222 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:24.214455 (+ 233us) server_negotiation.cc:407] Connection header received
0925 04:39:24.214641 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:24.214645 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:24.214710 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:24.214797 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:24.215535 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:24.216405 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:24.217529 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:24.217876 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:24.217971 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:24.220959 (+ 2988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:24.220964 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:24.220967 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:24.221002 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:24.222598 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:24.223072 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:24.223076 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:24.223078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:24.223163 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:24.223416 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:24.223419 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:24.223421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:24.223684 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:24.223812 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:24.224031 (+ 219us) server_negotiation.cc:299] Negotiation successful
0925 04:39:24.224151 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 04:39:24.227294 7928 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56480:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:39:24.227772 7928 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:39:24.232986 8066 tablet_service.cc:1467] Processing CreateTablet for tablet f13af5acc09a454fb1f9495e40dd0a32 (DEFAULT_TABLE table=test-table [id=9085bd6063824feea276a6a6f641b2cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:39:24.233012 8346 tablet_service.cc:1467] Processing CreateTablet for tablet f13af5acc09a454fb1f9495e40dd0a32 (DEFAULT_TABLE table=test-table [id=9085bd6063824feea276a6a6f641b2cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:39:24.233006 8206 tablet_service.cc:1467] Processing CreateTablet for tablet f13af5acc09a454fb1f9495e40dd0a32 (DEFAULT_TABLE table=test-table [id=9085bd6063824feea276a6a6f641b2cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:39:24.233304 8206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f13af5acc09a454fb1f9495e40dd0a32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:39:24.233304 8066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f13af5acc09a454fb1f9495e40dd0a32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:39:24.233304 8346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f13af5acc09a454fb1f9495e40dd0a32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:39:24.235706 8440 tablet_bootstrap.cc:492] T f13af5acc09a454fb1f9495e40dd0a32 P a310b4a5969a4b5686535c2d19fd3c24: Bootstrap starting.
I20260925 04:39:24.235762 8441 tablet_bootstrap.cc:492] T f13af5acc09a454fb1f9495e40dd0a32 P bf756d2a72414257ae7197d5ca26d0d7: Bootstrap starting.
I20260925 04:39:24.235785 8442 tablet_bootstrap.cc:492] T f13af5acc09a454fb1f9495e40dd0a32 P d73297192af149f3b757984ddd9afcb0: Bootstrap starting.
I20260925 04:39:24.236517 8442 tablet_bootstrap.cc:654] T f13af5acc09a454fb1f9495e40dd0a32 P d73297192af149f3b757984ddd9afcb0: Neither blocks nor log segments found. Creating new log.
I20260925 04:39:24.236572 8440 tablet_bootstrap.cc:654] T f13af5acc09a454fb1f9495e40dd0a32 P a310b4a5969a4b5686535c2d19fd3c24: Neither blocks nor log segments found. Creating new log.
I20260925 04:39:24.236627 8441 tablet_bootstrap.cc:654] T f13af5acc09a454fb1f9495e40dd0a32 P bf756d2a72414257ae7197d5ca26d0d7: Neither blocks nor log segments found. Creating new log.
I20260925 04:39:24.237803 8442 tablet_bootstrap.cc:492] T f13af5acc09a454fb1f9495e40dd0a32 P d73297192af149f3b757984ddd9afcb0: No bootstrap required, opened a new log
I20260925 04:39:24.237905 8442 ts_tablet_manager.cc:1397] T f13af5acc09a454fb1f9495e40dd0a32 P d73297192af149f3b757984ddd9afcb0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 04:39:24.238386 8441 tablet_bootstrap.cc:492] T f13af5acc09a454fb1f9495e40dd0a32 P bf756d2a72414257ae7197d5ca26d0d7: No bootstrap required, opened a new log
I20260925 04:39:24.238473 8441 ts_tablet_manager.cc:1397] T f13af5acc09a454fb1f9495e40dd0a32 P bf756d2a72414257ae7197d5ca26d0d7: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260925 04:39:24.238652 8442 raft_consensus.cc:348] T f13af5acc09a454fb1f9495e40dd0a32 P d73297192af149f3b757984ddd9afcb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73297192af149f3b757984ddd9afcb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33961 } } peers { permanent_uuid: "bf756d2a72414257ae7197d5ca26d0d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 } } peers { permanent_uuid: "a310b4a5969a4b5686535c2d19fd3c24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46153 } }
I20260925 04:39:24.238750 8442 raft_consensus.cc:374] T f13af5acc09a454fb1f9495e40dd0a32 P d73297192af149f3b757984ddd9afcb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:39:24.238770 8442 raft_consensus.cc:729] T f13af5acc09a454fb1f9495e40dd0a32 P d73297192af149f3b757984ddd9afcb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73297192af149f3b757984ddd9afcb0, State: Initialized, Role: FOLLOWER
I20260925 04:39:24.238876 8442 consensus_queue.cc:260] T f13af5acc09a454fb1f9495e40dd0a32 P d73297192af149f3b757984ddd9afcb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73297192af149f3b757984ddd9afcb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33961 } } peers { permanent_uuid: "bf756d2a72414257ae7197d5ca26d0d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 } } peers { permanent_uuid: "a310b4a5969a4b5686535c2d19fd3c24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46153 } }
I20260925 04:39:24.238919 8441 raft_consensus.cc:348] T f13af5acc09a454fb1f9495e40dd0a32 P bf756d2a72414257ae7197d5ca26d0d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73297192af149f3b757984ddd9afcb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33961 } } peers { permanent_uuid: "bf756d2a72414257ae7197d5ca26d0d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 } } peers { permanent_uuid: "a310b4a5969a4b5686535c2d19fd3c24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46153 } }
I20260925 04:39:24.239027 8441 raft_consensus.cc:374] T f13af5acc09a454fb1f9495e40dd0a32 P bf756d2a72414257ae7197d5ca26d0d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:39:24.239054 8441 raft_consensus.cc:729] T f13af5acc09a454fb1f9495e40dd0a32 P bf756d2a72414257ae7197d5ca26d0d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf756d2a72414257ae7197d5ca26d0d7, State: Initialized, Role: FOLLOWER
I20260925 04:39:24.239122 8442 ts_tablet_manager.cc:1428] T f13af5acc09a454fb1f9495e40dd0a32 P d73297192af149f3b757984ddd9afcb0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 04:39:24.239163 8441 consensus_queue.cc:260] T f13af5acc09a454fb1f9495e40dd0a32 P bf756d2a72414257ae7197d5ca26d0d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73297192af149f3b757984ddd9afcb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33961 } } peers { permanent_uuid: "bf756d2a72414257ae7197d5ca26d0d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 } } peers { permanent_uuid: "a310b4a5969a4b5686535c2d19fd3c24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46153 } }
I20260925 04:39:24.239393 8441 ts_tablet_manager.cc:1428] T f13af5acc09a454fb1f9495e40dd0a32 P bf756d2a72414257ae7197d5ca26d0d7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 04:39:24.240641 8440 tablet_bootstrap.cc:492] T f13af5acc09a454fb1f9495e40dd0a32 P a310b4a5969a4b5686535c2d19fd3c24: No bootstrap required, opened a new log
I20260925 04:39:24.240778 8440 ts_tablet_manager.cc:1397] T f13af5acc09a454fb1f9495e40dd0a32 P a310b4a5969a4b5686535c2d19fd3c24: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260925 04:39:24.241246 8440 raft_consensus.cc:348] T f13af5acc09a454fb1f9495e40dd0a32 P a310b4a5969a4b5686535c2d19fd3c24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73297192af149f3b757984ddd9afcb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33961 } } peers { permanent_uuid: "bf756d2a72414257ae7197d5ca26d0d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 } } peers { permanent_uuid: "a310b4a5969a4b5686535c2d19fd3c24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46153 } }
I20260925 04:39:24.241387 8440 raft_consensus.cc:374] T f13af5acc09a454fb1f9495e40dd0a32 P a310b4a5969a4b5686535c2d19fd3c24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:39:24.241441 8440 raft_consensus.cc:729] T f13af5acc09a454fb1f9495e40dd0a32 P a310b4a5969a4b5686535c2d19fd3c24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a310b4a5969a4b5686535c2d19fd3c24, State: Initialized, Role: FOLLOWER
I20260925 04:39:24.241581 8440 consensus_queue.cc:260] T f13af5acc09a454fb1f9495e40dd0a32 P a310b4a5969a4b5686535c2d19fd3c24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73297192af149f3b757984ddd9afcb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33961 } } peers { permanent_uuid: "bf756d2a72414257ae7197d5ca26d0d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 } } peers { permanent_uuid: "a310b4a5969a4b5686535c2d19fd3c24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46153 } }
I20260925 04:39:24.241848 8440 ts_tablet_manager.cc:1428] T f13af5acc09a454fb1f9495e40dd0a32 P a310b4a5969a4b5686535c2d19fd3c24: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 04:39:24.368149 8450 raft_consensus.cc:482] T f13af5acc09a454fb1f9495e40dd0a32 P a310b4a5969a4b5686535c2d19fd3c24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:39:24.368302 8450 raft_consensus.cc:504] T f13af5acc09a454fb1f9495e40dd0a32 P a310b4a5969a4b5686535c2d19fd3c24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73297192af149f3b757984ddd9afcb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33961 } } peers { permanent_uuid: "bf756d2a72414257ae7197d5ca26d0d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 } } peers { permanent_uuid: "a310b4a5969a4b5686535c2d19fd3c24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46153 } }
I20260925 04:39:24.368774 8450 leader_election.cc:288] T f13af5acc09a454fb1f9495e40dd0a32 P a310b4a5969a4b5686535c2d19fd3c24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d73297192af149f3b757984ddd9afcb0 (127.0.98.1:33961), bf756d2a72414257ae7197d5ca26d0d7 (127.0.98.2:42389)
I20260925 04:39:24.369292 8086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f13af5acc09a454fb1f9495e40dd0a32" candidate_uuid: "a310b4a5969a4b5686535c2d19fd3c24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73297192af149f3b757984ddd9afcb0" is_pre_election: true
I20260925 04:39:24.369292 8226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f13af5acc09a454fb1f9495e40dd0a32" candidate_uuid: "a310b4a5969a4b5686535c2d19fd3c24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf756d2a72414257ae7197d5ca26d0d7" is_pre_election: true
I20260925 04:39:24.369462 8086 raft_consensus.cc:2454] T f13af5acc09a454fb1f9495e40dd0a32 P d73297192af149f3b757984ddd9afcb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a310b4a5969a4b5686535c2d19fd3c24 in term 0.
I20260925 04:39:24.369462 8226 raft_consensus.cc:2454] T f13af5acc09a454fb1f9495e40dd0a32 P bf756d2a72414257ae7197d5ca26d0d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a310b4a5969a4b5686535c2d19fd3c24 in term 0.
I20260925 04:39:24.369810 8300 leader_election.cc:302] T f13af5acc09a454fb1f9495e40dd0a32 P a310b4a5969a4b5686535c2d19fd3c24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: a310b4a5969a4b5686535c2d19fd3c24, bf756d2a72414257ae7197d5ca26d0d7, d73297192af149f3b757984ddd9afcb0; no voters:
I20260925 04:39:24.370005 8450 raft_consensus.cc:2781] T f13af5acc09a454fb1f9495e40dd0a32 P a310b4a5969a4b5686535c2d19fd3c24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:39:24.370070 8450 raft_consensus.cc:482] T f13af5acc09a454fb1f9495e40dd0a32 P a310b4a5969a4b5686535c2d19fd3c24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:39:24.370100 8450 raft_consensus.cc:3037] T f13af5acc09a454fb1f9495e40dd0a32 P a310b4a5969a4b5686535c2d19fd3c24 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:39:24.371141 8450 raft_consensus.cc:504] T f13af5acc09a454fb1f9495e40dd0a32 P a310b4a5969a4b5686535c2d19fd3c24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73297192af149f3b757984ddd9afcb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33961 } } peers { permanent_uuid: "bf756d2a72414257ae7197d5ca26d0d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 } } peers { permanent_uuid: "a310b4a5969a4b5686535c2d19fd3c24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46153 } }
I20260925 04:39:24.371500 8450 leader_election.cc:288] T f13af5acc09a454fb1f9495e40dd0a32 P a310b4a5969a4b5686535c2d19fd3c24 [CANDIDATE]: Term 1 election: Requested vote from peers d73297192af149f3b757984ddd9afcb0 (127.0.98.1:33961), bf756d2a72414257ae7197d5ca26d0d7 (127.0.98.2:42389)
I20260925 04:39:24.371930 8086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f13af5acc09a454fb1f9495e40dd0a32" candidate_uuid: "a310b4a5969a4b5686535c2d19fd3c24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73297192af149f3b757984ddd9afcb0"
I20260925 04:39:24.371928 8226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f13af5acc09a454fb1f9495e40dd0a32" candidate_uuid: "a310b4a5969a4b5686535c2d19fd3c24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf756d2a72414257ae7197d5ca26d0d7"
I20260925 04:39:24.372071 8086 raft_consensus.cc:3037] T f13af5acc09a454fb1f9495e40dd0a32 P d73297192af149f3b757984ddd9afcb0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:39:24.372069 8226 raft_consensus.cc:3037] T f13af5acc09a454fb1f9495e40dd0a32 P bf756d2a72414257ae7197d5ca26d0d7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:39:24.372992 8226 raft_consensus.cc:2454] T f13af5acc09a454fb1f9495e40dd0a32 P bf756d2a72414257ae7197d5ca26d0d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a310b4a5969a4b5686535c2d19fd3c24 in term 1.
I20260925 04:39:24.372994 8086 raft_consensus.cc:2454] T f13af5acc09a454fb1f9495e40dd0a32 P d73297192af149f3b757984ddd9afcb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a310b4a5969a4b5686535c2d19fd3c24 in term 1.
I20260925 04:39:24.373294 8300 leader_election.cc:302] T f13af5acc09a454fb1f9495e40dd0a32 P a310b4a5969a4b5686535c2d19fd3c24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a310b4a5969a4b5686535c2d19fd3c24, d73297192af149f3b757984ddd9afcb0; no voters:
I20260925 04:39:24.373473 8450 raft_consensus.cc:2781] T f13af5acc09a454fb1f9495e40dd0a32 P a310b4a5969a4b5686535c2d19fd3c24 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:39:24.373569 8450 raft_consensus.cc:686] T f13af5acc09a454fb1f9495e40dd0a32 P a310b4a5969a4b5686535c2d19fd3c24 [term 1 LEADER]: Becoming Leader. State: Replica: a310b4a5969a4b5686535c2d19fd3c24, State: Running, Role: LEADER
I20260925 04:39:24.373710 8450 consensus_queue.cc:237] T f13af5acc09a454fb1f9495e40dd0a32 P a310b4a5969a4b5686535c2d19fd3c24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73297192af149f3b757984ddd9afcb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33961 } } peers { permanent_uuid: "bf756d2a72414257ae7197d5ca26d0d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 } } peers { permanent_uuid: "a310b4a5969a4b5686535c2d19fd3c24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46153 } }
I20260925 04:39:24.375438 7928 catalog_manager.cc:5495] T f13af5acc09a454fb1f9495e40dd0a32 P a310b4a5969a4b5686535c2d19fd3c24 reported cstate change: term changed from 0 to 1, leader changed from <none> to a310b4a5969a4b5686535c2d19fd3c24 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "a310b4a5969a4b5686535c2d19fd3c24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73297192af149f3b757984ddd9afcb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf756d2a72414257ae7197d5ca26d0d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a310b4a5969a4b5686535c2d19fd3c24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46153 } health_report { overall_health: HEALTHY } } }
I20260925 04:39:24.398136 8438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:24.395179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39090 (local address 127.0.98.3:46153)
0925 04:39:24.395299 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:24.395304 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:24.395319 (+ 15us) server_negotiation.cc:407] Connection header received
0925 04:39:24.395395 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:24.395398 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:24.395442 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:24.395519 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:39:24.395793 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:24.396299 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:24.397143 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:24.397358 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:24.397444 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:24.397699 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:39:24.397775 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:39:24.397831 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:24.397935 (+ 104us) server_negotiation.cc:299] Negotiation successful
0925 04:39:24.397987 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 04:39:24.400187 8086 raft_consensus.cc:1261] T f13af5acc09a454fb1f9495e40dd0a32 P d73297192af149f3b757984ddd9afcb0 [term 1 FOLLOWER]: Refusing update from remote peer a310b4a5969a4b5686535c2d19fd3c24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:39:24.400187 8226 raft_consensus.cc:1261] T f13af5acc09a454fb1f9495e40dd0a32 P bf756d2a72414257ae7197d5ca26d0d7 [term 1 FOLLOWER]: Refusing update from remote peer a310b4a5969a4b5686535c2d19fd3c24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:39:24.400643 8447 consensus_queue.cc:1035] T f13af5acc09a454fb1f9495e40dd0a32 P a310b4a5969a4b5686535c2d19fd3c24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf756d2a72414257ae7197d5ca26d0d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:39:24.400815 8450 consensus_queue.cc:1035] T f13af5acc09a454fb1f9495e40dd0a32 P a310b4a5969a4b5686535c2d19fd3c24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73297192af149f3b757984ddd9afcb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:39:24.408147 7929 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:39:24.408283 7929 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:39:24.410081 7929 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bf7d385c164047fd9a6fc095948982f6: Sending DeleteTablet for 3 replicas of tablet f13af5acc09a454fb1f9495e40dd0a32
I20260925 04:39:24.410640 8346 tablet_service.cc:1514] Processing DeleteTablet for tablet f13af5acc09a454fb1f9495e40dd0a32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:39:24 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39064
I20260925 04:39:24.410640 8206 tablet_service.cc:1514] Processing DeleteTablet for tablet f13af5acc09a454fb1f9495e40dd0a32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:39:24 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49130
I20260925 04:39:24.410843 8464 tablet_replica.cc:330] T f13af5acc09a454fb1f9495e40dd0a32 P a310b4a5969a4b5686535c2d19fd3c24: stopping tablet replica
I20260925 04:39:24.410841 8066 tablet_service.cc:1514] Processing DeleteTablet for tablet f13af5acc09a454fb1f9495e40dd0a32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:39:24 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43978
I20260925 04:39:24.410959 8464 raft_consensus.cc:2229] T f13af5acc09a454fb1f9495e40dd0a32 P a310b4a5969a4b5686535c2d19fd3c24 [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:39:24.411013 8463 tablet_replica.cc:330] T f13af5acc09a454fb1f9495e40dd0a32 P d73297192af149f3b757984ddd9afcb0: stopping tablet replica
I20260925 04:39:24.411149 8463 raft_consensus.cc:2229] T f13af5acc09a454fb1f9495e40dd0a32 P d73297192af149f3b757984ddd9afcb0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:39:24.411183 8464 raft_consensus.cc:2258] T f13af5acc09a454fb1f9495e40dd0a32 P a310b4a5969a4b5686535c2d19fd3c24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:39:24.411279 8463 raft_consensus.cc:2258] T f13af5acc09a454fb1f9495e40dd0a32 P d73297192af149f3b757984ddd9afcb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:39:24.411360 8462 tablet_replica.cc:330] T f13af5acc09a454fb1f9495e40dd0a32 P bf756d2a72414257ae7197d5ca26d0d7: stopping tablet replica
I20260925 04:39:24.411466 8462 raft_consensus.cc:2229] T f13af5acc09a454fb1f9495e40dd0a32 P bf756d2a72414257ae7197d5ca26d0d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:39:24.411587 8462 raft_consensus.cc:2258] T f13af5acc09a454fb1f9495e40dd0a32 P bf756d2a72414257ae7197d5ca26d0d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:39:24.412027 8464 ts_tablet_manager.cc:1905] T f13af5acc09a454fb1f9495e40dd0a32 P a310b4a5969a4b5686535c2d19fd3c24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:39:24.412062 8463 ts_tablet_manager.cc:1905] T f13af5acc09a454fb1f9495e40dd0a32 P d73297192af149f3b757984ddd9afcb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:39:24.412051 7929 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56480:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:39:24.412453 7929 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:39:24.413532 8462 ts_tablet_manager.cc:1905] T f13af5acc09a454fb1f9495e40dd0a32 P bf756d2a72414257ae7197d5ca26d0d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:39:24.414522 8464 ts_tablet_manager.cc:1918] T f13af5acc09a454fb1f9495e40dd0a32 P a310b4a5969a4b5686535c2d19fd3c24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:39:24.414538 8463 ts_tablet_manager.cc:1918] T f13af5acc09a454fb1f9495e40dd0a32 P d73297192af149f3b757984ddd9afcb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:39:24.414601 8463 log.cc:1196] T f13af5acc09a454fb1f9495e40dd0a32 P d73297192af149f3b757984ddd9afcb0: Deleting WAL directory at /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/ts-0/wal/wals/f13af5acc09a454fb1f9495e40dd0a32
I20260925 04:39:24.414601 8464 log.cc:1196] T f13af5acc09a454fb1f9495e40dd0a32 P a310b4a5969a4b5686535c2d19fd3c24: Deleting WAL directory at /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/ts-2/wal/wals/f13af5acc09a454fb1f9495e40dd0a32
I20260925 04:39:24.414844 8464 ts_tablet_manager.cc:1939] T f13af5acc09a454fb1f9495e40dd0a32 P a310b4a5969a4b5686535c2d19fd3c24: Deleting consensus metadata
I20260925 04:39:24.414844 8463 ts_tablet_manager.cc:1939] T f13af5acc09a454fb1f9495e40dd0a32 P d73297192af149f3b757984ddd9afcb0: Deleting consensus metadata
I20260925 04:39:24.415922 8462 ts_tablet_manager.cc:1918] T f13af5acc09a454fb1f9495e40dd0a32 P bf756d2a72414257ae7197d5ca26d0d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:39:24.416005 8462 log.cc:1196] T f13af5acc09a454fb1f9495e40dd0a32 P bf756d2a72414257ae7197d5ca26d0d7: Deleting WAL directory at /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/ts-1/wal/wals/f13af5acc09a454fb1f9495e40dd0a32
I20260925 04:39:24.416235 8462 ts_tablet_manager.cc:1939] T f13af5acc09a454fb1f9495e40dd0a32 P bf756d2a72414257ae7197d5ca26d0d7: Deleting consensus metadata
I20260925 04:39:24.416303 7916 catalog_manager.cc:4841] TS d73297192af149f3b757984ddd9afcb0 (127.0.98.1:33961): tablet f13af5acc09a454fb1f9495e40dd0a32 (table test-table [id=9085bd6063824feea276a6a6f641b2cf]) successfully deleted
I20260925 04:39:24.416484 7916 catalog_manager.cc:4841] TS a310b4a5969a4b5686535c2d19fd3c24 (127.0.98.3:46153): tablet f13af5acc09a454fb1f9495e40dd0a32 (table test-table [id=9085bd6063824feea276a6a6f641b2cf]) successfully deleted
I20260925 04:39:24.416992 7913 catalog_manager.cc:4841] TS bf756d2a72414257ae7197d5ca26d0d7 (127.0.98.2:42389): tablet f13af5acc09a454fb1f9495e40dd0a32 (table test-table [id=9085bd6063824feea276a6a6f641b2cf]) successfully deleted
I20260925 04:39:24.417918 8066 tablet_service.cc:1467] Processing CreateTablet for tablet b5cd23c499104a569f3ece87c953ad3f (DEFAULT_TABLE table=test-table [id=ced11a437ba644e79e1f383c75393d41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:39:24.417946 8206 tablet_service.cc:1467] Processing CreateTablet for tablet b5cd23c499104a569f3ece87c953ad3f (DEFAULT_TABLE table=test-table [id=ced11a437ba644e79e1f383c75393d41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:39:24.418181 8206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5cd23c499104a569f3ece87c953ad3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:39:24.418200 8066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5cd23c499104a569f3ece87c953ad3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:39:24.418171 8346 tablet_service.cc:1467] Processing CreateTablet for tablet b5cd23c499104a569f3ece87c953ad3f (DEFAULT_TABLE table=test-table [id=ced11a437ba644e79e1f383c75393d41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:39:24.418409 8346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5cd23c499104a569f3ece87c953ad3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:39:24.419992 8441 tablet_bootstrap.cc:492] T b5cd23c499104a569f3ece87c953ad3f P bf756d2a72414257ae7197d5ca26d0d7: Bootstrap starting.
I20260925 04:39:24.420591 8442 tablet_bootstrap.cc:492] T b5cd23c499104a569f3ece87c953ad3f P d73297192af149f3b757984ddd9afcb0: Bootstrap starting.
I20260925 04:39:24.420718 8440 tablet_bootstrap.cc:492] T b5cd23c499104a569f3ece87c953ad3f P a310b4a5969a4b5686535c2d19fd3c24: Bootstrap starting.
I20260925 04:39:24.420818 8441 tablet_bootstrap.cc:654] T b5cd23c499104a569f3ece87c953ad3f P bf756d2a72414257ae7197d5ca26d0d7: Neither blocks nor log segments found. Creating new log.
I20260925 04:39:24.421509 8442 tablet_bootstrap.cc:654] T b5cd23c499104a569f3ece87c953ad3f P d73297192af149f3b757984ddd9afcb0: Neither blocks nor log segments found. Creating new log.
I20260925 04:39:24.422075 8440 tablet_bootstrap.cc:654] T b5cd23c499104a569f3ece87c953ad3f P a310b4a5969a4b5686535c2d19fd3c24: Neither blocks nor log segments found. Creating new log.
I20260925 04:39:24.422693 8442 tablet_bootstrap.cc:492] T b5cd23c499104a569f3ece87c953ad3f P d73297192af149f3b757984ddd9afcb0: No bootstrap required, opened a new log
I20260925 04:39:24.422783 8442 ts_tablet_manager.cc:1397] T b5cd23c499104a569f3ece87c953ad3f P d73297192af149f3b757984ddd9afcb0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 04:39:24.423269 8442 raft_consensus.cc:348] T b5cd23c499104a569f3ece87c953ad3f P d73297192af149f3b757984ddd9afcb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73297192af149f3b757984ddd9afcb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33961 } } peers { permanent_uuid: "bf756d2a72414257ae7197d5ca26d0d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 } } peers { permanent_uuid: "a310b4a5969a4b5686535c2d19fd3c24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46153 } }
I20260925 04:39:24.423390 8442 raft_consensus.cc:374] T b5cd23c499104a569f3ece87c953ad3f P d73297192af149f3b757984ddd9afcb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:39:24.423424 8442 raft_consensus.cc:729] T b5cd23c499104a569f3ece87c953ad3f P d73297192af149f3b757984ddd9afcb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73297192af149f3b757984ddd9afcb0, State: Initialized, Role: FOLLOWER
I20260925 04:39:24.423554 8442 consensus_queue.cc:260] T b5cd23c499104a569f3ece87c953ad3f P d73297192af149f3b757984ddd9afcb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73297192af149f3b757984ddd9afcb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33961 } } peers { permanent_uuid: "bf756d2a72414257ae7197d5ca26d0d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 } } peers { permanent_uuid: "a310b4a5969a4b5686535c2d19fd3c24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46153 } }
I20260925 04:39:24.423856 8442 ts_tablet_manager.cc:1428] T b5cd23c499104a569f3ece87c953ad3f P d73297192af149f3b757984ddd9afcb0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 04:39:24.424115 8440 tablet_bootstrap.cc:492] T b5cd23c499104a569f3ece87c953ad3f P a310b4a5969a4b5686535c2d19fd3c24: No bootstrap required, opened a new log
I20260925 04:39:24.424250 8440 ts_tablet_manager.cc:1397] T b5cd23c499104a569f3ece87c953ad3f P a310b4a5969a4b5686535c2d19fd3c24: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260925 04:39:24.424381 8441 tablet_bootstrap.cc:492] T b5cd23c499104a569f3ece87c953ad3f P bf756d2a72414257ae7197d5ca26d0d7: No bootstrap required, opened a new log
I20260925 04:39:24.424463 8441 ts_tablet_manager.cc:1397] T b5cd23c499104a569f3ece87c953ad3f P bf756d2a72414257ae7197d5ca26d0d7: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260925 04:39:24.424717 8440 raft_consensus.cc:348] T b5cd23c499104a569f3ece87c953ad3f P a310b4a5969a4b5686535c2d19fd3c24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73297192af149f3b757984ddd9afcb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33961 } } peers { permanent_uuid: "bf756d2a72414257ae7197d5ca26d0d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 } } peers { permanent_uuid: "a310b4a5969a4b5686535c2d19fd3c24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46153 } }
I20260925 04:39:24.424824 8440 raft_consensus.cc:374] T b5cd23c499104a569f3ece87c953ad3f P a310b4a5969a4b5686535c2d19fd3c24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:39:24.424853 8440 raft_consensus.cc:729] T b5cd23c499104a569f3ece87c953ad3f P a310b4a5969a4b5686535c2d19fd3c24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a310b4a5969a4b5686535c2d19fd3c24, State: Initialized, Role: FOLLOWER
I20260925 04:39:24.424853 8441 raft_consensus.cc:348] T b5cd23c499104a569f3ece87c953ad3f P bf756d2a72414257ae7197d5ca26d0d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73297192af149f3b757984ddd9afcb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33961 } } peers { permanent_uuid: "bf756d2a72414257ae7197d5ca26d0d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 } } peers { permanent_uuid: "a310b4a5969a4b5686535c2d19fd3c24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46153 } }
I20260925 04:39:24.424976 8441 raft_consensus.cc:374] T b5cd23c499104a569f3ece87c953ad3f P bf756d2a72414257ae7197d5ca26d0d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:39:24.425020 8441 raft_consensus.cc:729] T b5cd23c499104a569f3ece87c953ad3f P bf756d2a72414257ae7197d5ca26d0d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf756d2a72414257ae7197d5ca26d0d7, State: Initialized, Role: FOLLOWER
I20260925 04:39:24.424975 8440 consensus_queue.cc:260] T b5cd23c499104a569f3ece87c953ad3f P a310b4a5969a4b5686535c2d19fd3c24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73297192af149f3b757984ddd9afcb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33961 } } peers { permanent_uuid: "bf756d2a72414257ae7197d5ca26d0d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 } } peers { permanent_uuid: "a310b4a5969a4b5686535c2d19fd3c24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46153 } }
I20260925 04:39:24.425156 8441 consensus_queue.cc:260] T b5cd23c499104a569f3ece87c953ad3f P bf756d2a72414257ae7197d5ca26d0d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73297192af149f3b757984ddd9afcb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33961 } } peers { permanent_uuid: "bf756d2a72414257ae7197d5ca26d0d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 } } peers { permanent_uuid: "a310b4a5969a4b5686535c2d19fd3c24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46153 } }
I20260925 04:39:24.425222 8440 ts_tablet_manager.cc:1428] T b5cd23c499104a569f3ece87c953ad3f P a310b4a5969a4b5686535c2d19fd3c24: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 04:39:24.425418 8441 ts_tablet_manager.cc:1428] T b5cd23c499104a569f3ece87c953ad3f P bf756d2a72414257ae7197d5ca26d0d7: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260925 04:39:24.481581 8450 raft_consensus.cc:482] T b5cd23c499104a569f3ece87c953ad3f P a310b4a5969a4b5686535c2d19fd3c24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:39:24.481743 8450 raft_consensus.cc:504] T b5cd23c499104a569f3ece87c953ad3f P a310b4a5969a4b5686535c2d19fd3c24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73297192af149f3b757984ddd9afcb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33961 } } peers { permanent_uuid: "bf756d2a72414257ae7197d5ca26d0d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 } } peers { permanent_uuid: "a310b4a5969a4b5686535c2d19fd3c24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46153 } }
I20260925 04:39:24.482173 8450 leader_election.cc:288] T b5cd23c499104a569f3ece87c953ad3f P a310b4a5969a4b5686535c2d19fd3c24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d73297192af149f3b757984ddd9afcb0 (127.0.98.1:33961), bf756d2a72414257ae7197d5ca26d0d7 (127.0.98.2:42389)
I20260925 04:39:24.482651 8086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5cd23c499104a569f3ece87c953ad3f" candidate_uuid: "a310b4a5969a4b5686535c2d19fd3c24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73297192af149f3b757984ddd9afcb0" is_pre_election: true
I20260925 04:39:24.482649 8226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5cd23c499104a569f3ece87c953ad3f" candidate_uuid: "a310b4a5969a4b5686535c2d19fd3c24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf756d2a72414257ae7197d5ca26d0d7" is_pre_election: true
I20260925 04:39:24.482810 8086 raft_consensus.cc:2454] T b5cd23c499104a569f3ece87c953ad3f P d73297192af149f3b757984ddd9afcb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a310b4a5969a4b5686535c2d19fd3c24 in term 0.
I20260925 04:39:24.482810 8226 raft_consensus.cc:2454] T b5cd23c499104a569f3ece87c953ad3f P bf756d2a72414257ae7197d5ca26d0d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a310b4a5969a4b5686535c2d19fd3c24 in term 0.
I20260925 04:39:24.483115 8297 leader_election.cc:302] T b5cd23c499104a569f3ece87c953ad3f P a310b4a5969a4b5686535c2d19fd3c24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a310b4a5969a4b5686535c2d19fd3c24, bf756d2a72414257ae7197d5ca26d0d7; no voters:
I20260925 04:39:24.483287 8450 raft_consensus.cc:2781] T b5cd23c499104a569f3ece87c953ad3f P a310b4a5969a4b5686535c2d19fd3c24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:39:24.483348 8450 raft_consensus.cc:482] T b5cd23c499104a569f3ece87c953ad3f P a310b4a5969a4b5686535c2d19fd3c24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:39:24.483379 8450 raft_consensus.cc:3037] T b5cd23c499104a569f3ece87c953ad3f P a310b4a5969a4b5686535c2d19fd3c24 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:39:24.484531 8450 raft_consensus.cc:504] T b5cd23c499104a569f3ece87c953ad3f P a310b4a5969a4b5686535c2d19fd3c24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73297192af149f3b757984ddd9afcb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33961 } } peers { permanent_uuid: "bf756d2a72414257ae7197d5ca26d0d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 } } peers { permanent_uuid: "a310b4a5969a4b5686535c2d19fd3c24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46153 } }
I20260925 04:39:24.484879 8450 leader_election.cc:288] T b5cd23c499104a569f3ece87c953ad3f P a310b4a5969a4b5686535c2d19fd3c24 [CANDIDATE]: Term 1 election: Requested vote from peers d73297192af149f3b757984ddd9afcb0 (127.0.98.1:33961), bf756d2a72414257ae7197d5ca26d0d7 (127.0.98.2:42389)
I20260925 04:39:24.485255 8086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5cd23c499104a569f3ece87c953ad3f" candidate_uuid: "a310b4a5969a4b5686535c2d19fd3c24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73297192af149f3b757984ddd9afcb0"
I20260925 04:39:24.485250 8226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5cd23c499104a569f3ece87c953ad3f" candidate_uuid: "a310b4a5969a4b5686535c2d19fd3c24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf756d2a72414257ae7197d5ca26d0d7"
I20260925 04:39:24.485388 8226 raft_consensus.cc:3037] T b5cd23c499104a569f3ece87c953ad3f P bf756d2a72414257ae7197d5ca26d0d7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:39:24.485388 8086 raft_consensus.cc:3037] T b5cd23c499104a569f3ece87c953ad3f P d73297192af149f3b757984ddd9afcb0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:39:24.486495 8086 raft_consensus.cc:2454] T b5cd23c499104a569f3ece87c953ad3f P d73297192af149f3b757984ddd9afcb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a310b4a5969a4b5686535c2d19fd3c24 in term 1.
I20260925 04:39:24.486495 8226 raft_consensus.cc:2454] T b5cd23c499104a569f3ece87c953ad3f P bf756d2a72414257ae7197d5ca26d0d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a310b4a5969a4b5686535c2d19fd3c24 in term 1.
I20260925 04:39:24.486814 8300 leader_election.cc:302] T b5cd23c499104a569f3ece87c953ad3f P a310b4a5969a4b5686535c2d19fd3c24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: a310b4a5969a4b5686535c2d19fd3c24, bf756d2a72414257ae7197d5ca26d0d7, d73297192af149f3b757984ddd9afcb0; no voters:
I20260925 04:39:24.487027 8450 raft_consensus.cc:2781] T b5cd23c499104a569f3ece87c953ad3f P a310b4a5969a4b5686535c2d19fd3c24 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:39:24.487121 8450 raft_consensus.cc:686] T b5cd23c499104a569f3ece87c953ad3f P a310b4a5969a4b5686535c2d19fd3c24 [term 1 LEADER]: Becoming Leader. State: Replica: a310b4a5969a4b5686535c2d19fd3c24, State: Running, Role: LEADER
I20260925 04:39:24.487282 8450 consensus_queue.cc:237] T b5cd23c499104a569f3ece87c953ad3f P a310b4a5969a4b5686535c2d19fd3c24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73297192af149f3b757984ddd9afcb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33961 } } peers { permanent_uuid: "bf756d2a72414257ae7197d5ca26d0d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 } } peers { permanent_uuid: "a310b4a5969a4b5686535c2d19fd3c24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46153 } }
I20260925 04:39:24.488950 7928 catalog_manager.cc:5495] T b5cd23c499104a569f3ece87c953ad3f P a310b4a5969a4b5686535c2d19fd3c24 reported cstate change: term changed from 0 to 1, leader changed from <none> to a310b4a5969a4b5686535c2d19fd3c24 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "a310b4a5969a4b5686535c2d19fd3c24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73297192af149f3b757984ddd9afcb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf756d2a72414257ae7197d5ca26d0d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a310b4a5969a4b5686535c2d19fd3c24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46153 } health_report { overall_health: HEALTHY } } }
I20260925 04:39:24.521390 8438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:24.518492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59065 (local address 127.0.98.3:46153)
0925 04:39:24.518596 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:24.518600 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:24.518657 (+ 57us) server_negotiation.cc:407] Connection header received
0925 04:39:24.518743 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:24.518746 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:24.518782 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:24.518840 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:39:24.519118 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:24.519635 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:24.520420 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:24.520628 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:24.520709 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:24.520949 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:39:24.521020 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:39:24.521075 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:24.521185 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 04:39:24.521229 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":49}
I20260925 04:39:24.521390 8495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:24.518417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46153 (local address 127.0.98.3:59065)
0925 04:39:24.518543 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:24.518559 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:24.518647 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:24.518856 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:24.518859 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:24.518865 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 04:39:24.519020 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:24.519027 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:24.519730 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:24.519733 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:24.520317 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:24.520325 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:24.520722 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:24.520725 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:24.520828 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:24.520850 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 04:39:24.521106 (+ 256us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 04:39:24.521111 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 04:39:24.521167 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 04:39:24.521250 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":20}
I20260925 04:39:24.526892 8226 raft_consensus.cc:1261] T b5cd23c499104a569f3ece87c953ad3f P bf756d2a72414257ae7197d5ca26d0d7 [term 1 FOLLOWER]: Refusing update from remote peer a310b4a5969a4b5686535c2d19fd3c24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:39:24.526932 8086 raft_consensus.cc:1261] T b5cd23c499104a569f3ece87c953ad3f P d73297192af149f3b757984ddd9afcb0 [term 1 FOLLOWER]: Refusing update from remote peer a310b4a5969a4b5686535c2d19fd3c24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:39:24.527256 8487 consensus_queue.cc:1035] T b5cd23c499104a569f3ece87c953ad3f P a310b4a5969a4b5686535c2d19fd3c24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf756d2a72414257ae7197d5ca26d0d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:39:24.527406 8450 consensus_queue.cc:1035] T b5cd23c499104a569f3ece87c953ad3f P a310b4a5969a4b5686535c2d19fd3c24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73297192af149f3b757984ddd9afcb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:39:24.560209 7928 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:39:24.560360 7928 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:39:24.562376 7928 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bf7d385c164047fd9a6fc095948982f6: Sending DeleteTablet for 3 replicas of tablet b5cd23c499104a569f3ece87c953ad3f
I20260925 04:39:24.562983 8206 tablet_service.cc:1514] Processing DeleteTablet for tablet b5cd23c499104a569f3ece87c953ad3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:39:24 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49130
I20260925 04:39:24.563057 8066 tablet_service.cc:1514] Processing DeleteTablet for tablet b5cd23c499104a569f3ece87c953ad3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:39:24 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43978
I20260925 04:39:24.563175 8462 tablet_replica.cc:330] T b5cd23c499104a569f3ece87c953ad3f P bf756d2a72414257ae7197d5ca26d0d7: stopping tablet replica
I20260925 04:39:24.563233 8463 tablet_replica.cc:330] T b5cd23c499104a569f3ece87c953ad3f P d73297192af149f3b757984ddd9afcb0: stopping tablet replica
I20260925 04:39:24.563295 8462 raft_consensus.cc:2229] T b5cd23c499104a569f3ece87c953ad3f P bf756d2a72414257ae7197d5ca26d0d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:39:24.563328 8463 raft_consensus.cc:2229] T b5cd23c499104a569f3ece87c953ad3f P d73297192af149f3b757984ddd9afcb0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:39:24.563417 8462 raft_consensus.cc:2258] T b5cd23c499104a569f3ece87c953ad3f P bf756d2a72414257ae7197d5ca26d0d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:39:24.563434 8463 raft_consensus.cc:2258] T b5cd23c499104a569f3ece87c953ad3f P d73297192af149f3b757984ddd9afcb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:39:24.563839 8345 tablet_service.cc:1514] Processing DeleteTablet for tablet b5cd23c499104a569f3ece87c953ad3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:39:24 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39064
I20260925 04:39:24.564024 8464 tablet_replica.cc:330] T b5cd23c499104a569f3ece87c953ad3f P a310b4a5969a4b5686535c2d19fd3c24: stopping tablet replica
I20260925 04:39:24.564131 8464 raft_consensus.cc:2229] T b5cd23c499104a569f3ece87c953ad3f P a310b4a5969a4b5686535c2d19fd3c24 [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:39:24.564297 8462 ts_tablet_manager.cc:1905] T b5cd23c499104a569f3ece87c953ad3f P bf756d2a72414257ae7197d5ca26d0d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:39:24.564321 8464 raft_consensus.cc:2258] T b5cd23c499104a569f3ece87c953ad3f P a310b4a5969a4b5686535c2d19fd3c24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:39:24.564324 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 8001
I20260925 04:39:24.566514 8462 ts_tablet_manager.cc:1918] T b5cd23c499104a569f3ece87c953ad3f P bf756d2a72414257ae7197d5ca26d0d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 04:39:24.566601 8462 log.cc:1196] T b5cd23c499104a569f3ece87c953ad3f P bf756d2a72414257ae7197d5ca26d0d7: Deleting WAL directory at /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/ts-1/wal/wals/b5cd23c499104a569f3ece87c953ad3f
I20260925 04:39:24.566800 8462 ts_tablet_manager.cc:1939] T b5cd23c499104a569f3ece87c953ad3f P bf756d2a72414257ae7197d5ca26d0d7: Deleting consensus metadata
I20260925 04:39:24.566973 8464 ts_tablet_manager.cc:1905] T b5cd23c499104a569f3ece87c953ad3f P a310b4a5969a4b5686535c2d19fd3c24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:39:24.567765 7913 catalog_manager.cc:4841] TS bf756d2a72414257ae7197d5ca26d0d7 (127.0.98.2:42389): tablet b5cd23c499104a569f3ece87c953ad3f (table test-table [id=ced11a437ba644e79e1f383c75393d41]) successfully deleted
I20260925 04:39:24.569165 8464 ts_tablet_manager.cc:1918] T b5cd23c499104a569f3ece87c953ad3f P a310b4a5969a4b5686535c2d19fd3c24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 04:39:24.569238 8464 log.cc:1196] T b5cd23c499104a569f3ece87c953ad3f P a310b4a5969a4b5686535c2d19fd3c24: Deleting WAL directory at /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311123478858-392-0/minicluster-data/ts-2/wal/wals/b5cd23c499104a569f3ece87c953ad3f
I20260925 04:39:24.569442 8464 ts_tablet_manager.cc:1939] T b5cd23c499104a569f3ece87c953ad3f P a310b4a5969a4b5686535c2d19fd3c24: Deleting consensus metadata
I20260925 04:39:24.570115 7916 catalog_manager.cc:4841] TS a310b4a5969a4b5686535c2d19fd3c24 (127.0.98.3:46153): tablet b5cd23c499104a569f3ece87c953ad3f (table test-table [id=ced11a437ba644e79e1f383c75393d41]) successfully deleted
W20260925 04:39:24.571197 7916 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33961 (error 108)
I20260925 04:39:24.571921 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 8141
I20260925 04:39:24.576879 8527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:24.571997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33961 (local address 127.0.0.1:43992)
0925 04:39:24.576650 (+ 4653us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:24.576757 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4550,"thread_start_us":49,"threads_started":1}
W20260925 04:39:24.577035 7916 catalog_manager.cc:4584] TS d73297192af149f3b757984ddd9afcb0 (127.0.98.1:33961): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b5cd23c499104a569f3ece87c953ad3f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33961: connect: Connection refused (error 111)
I20260925 04:39:24.578816 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 8281
I20260925 04:39:24.584896 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 7897
2026-09-25T04:39:24Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3053 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790311123478858-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:39:24 dist-test-slave-dc95 krb5kdc[8530](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:39:24 dist-test-slave-dc95 krb5kdc[8530](info): set up 2 sockets
Sep 25 04:39:24 dist-test-slave-dc95 krb5kdc[8530](info): commencing operation
krb5kdc: starting...
W20260925 04:39:25.552104 392 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790311123478858-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790311123478858-392-0/krb5kdc/test-user.keytab.
Sep 25 04:39:25 dist-test-slave-dc95 krb5kdc[8530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:39:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:39:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 04:39:25.675125 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790311123478858-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790311123478858-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790311123478858-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35427
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35427
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:39:25.787127 8543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:39:25.787436 8543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:39:25.787478 8543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:39:25.791756 8543 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:39:25.791839 8543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:39:25.791858 8543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:39:25.791874 8543 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:39:25.791889 8543 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:39:25.797927 8543 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790311123478858-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790311123478858-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35427
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790311123478858-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:39:25.799166 8543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:39:25.800259 8543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:39:25.809623 8551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:25.809623 8548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:25.809618 8549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:25.810024 8543 server_base.cc:1034] running on GCE node
I20260925 04:39:25.810644 8543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:39:25.812059 8543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:39:25.813308 8543 hybrid_clock.cc:648] HybridClock initialized: now 1790311165813272 us; error 39 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-25T04:39:25Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1238 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:39:25 dist-test-slave-dc95 krb5kdc[8556](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:39:25 dist-test-slave-dc95 krb5kdc[8556](info): set up 2 sockets
Sep 25 04:39:25 dist-test-slave-dc95 krb5kdc[8556](info): commencing operation
krb5kdc: starting...
W20260925 04:39:26.776423 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/krb5kdc/test-user.keytab.
Sep 25 04:39:26 dist-test-slave-dc95 krb5kdc[8556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:39:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:39:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 04:39:26.899226 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46371
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46371
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:39:27.011112 8569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:39:27.011413 8569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:39:27.011473 8569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:39:27.015781 8569 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:39:27.015868 8569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:39:27.015897 8569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:39:27.015918 8569 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:39:27.015939 8569 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:39:27.021678 8569 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46371
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:39:27.022881 8569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:39:27.024009 8569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:39:27.032933 8574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:27.032939 8575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:27.033155 8569 server_base.cc:1034] running on GCE node
W20260925 04:39:27.032939 8577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:27.033991 8569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:39:27.035223 8569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:39:27.036412 8569 hybrid_clock.cc:648] HybridClock initialized: now 1790311167036393 us; error 38 us; skew 500 ppm
Sep 25 04:39:27 dist-test-slave-dc95 krb5kdc[8556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311167, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:39:27.039435 8569 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 04:39:27.040469 8569 webserver.cc:460] Webserver started at http://127.0.98.62:37071/ using document root <none> and password file <none>
I20260925 04:39:27.041055 8569 fs_manager.cc:359] Metadata directory not provided
I20260925 04:39:27.041122 8569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:39:27.041344 8569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:39:27.043541 8569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/master-0/data/instance:
uuid: "33070ba178534b85bca652c3290138c9"
format_stamp: "Formatted at 2026-09-25 04:39:27 on dist-test-slave-dc95"
server_key: "e5a47f49f50d2f019843bfcce8459467"
server_key_iv: "db06888b9c35de5281e81750d1506fcf"
server_key_version: "encryptionkey@0"
I20260925 04:39:27.044196 8569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/master-0/wal/instance:
uuid: "33070ba178534b85bca652c3290138c9"
format_stamp: "Formatted at 2026-09-25 04:39:27 on dist-test-slave-dc95"
server_key: "e5a47f49f50d2f019843bfcce8459467"
server_key_iv: "db06888b9c35de5281e81750d1506fcf"
server_key_version: "encryptionkey@0"
I20260925 04:39:27.048130 8569 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 04:39:27.050660 8584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:27.052013 8569 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:39:27.052124 8569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/master-0/wal
uuid: "33070ba178534b85bca652c3290138c9"
format_stamp: "Formatted at 2026-09-25 04:39:27 on dist-test-slave-dc95"
server_key: "e5a47f49f50d2f019843bfcce8459467"
server_key_iv: "db06888b9c35de5281e81750d1506fcf"
server_key_version: "encryptionkey@0"
I20260925 04:39:27.052238 8569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:39:27.073897 8569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:39:27.077190 8569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:39:27.077371 8569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:39:27.084179 8569 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46371
I20260925 04:39:27.084203 8636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46371 every 8 connection(s)
I20260925 04:39:27.085148 8569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
I20260925 04:39:27.086014 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 8569
I20260925 04:39:27.086119 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/master-0/wal/instance
I20260925 04:39:27.086380 392 external_mini_cluster.cc:1428] Setting key cf8e5563df27052bb26995e6c26fbe4d
I20260925 04:39:27.088323 8637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:39:27.094189 8637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33070ba178534b85bca652c3290138c9: Bootstrap starting.
Sep 25 04:39:27 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:39:27.096772 8637 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33070ba178534b85bca652c3290138c9: Neither blocks nor log segments found. Creating new log.
I20260925 04:39:27.097538 8637 log.cc:824] T 00000000000000000000000000000000 P 33070ba178534b85bca652c3290138c9: Log is configured to *not* fsync() on all Append() calls
I20260925 04:39:27.099531 8637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33070ba178534b85bca652c3290138c9: No bootstrap required, opened a new log
I20260925 04:39:27.100351 8640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:27.087778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51656 (local address 127.0.98.62:46371)
0925 04:39:27.088209 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:27.088224 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:27.088259 (+ 35us) server_negotiation.cc:407] Connection header received
0925 04:39:27.088930 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:27.088949 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:27.089303 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:27.089627 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:27.090388 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:27.091392 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:27.092279 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:27.092761 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:27.092908 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:27.095366 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:27.095372 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:27.095384 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:27.095415 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:27.097346 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:27.097910 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:27.097915 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:27.097921 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:27.098045 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:27.098313 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:27.098316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:27.098317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:27.098738 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:27.098919 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:27.099217 (+ 298us) server_negotiation.cc:299] Negotiation successful
0925 04:39:27.099530 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":155,"threads_started":1}
I20260925 04:39:27.103377 8637 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33070ba178534b85bca652c3290138c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33070ba178534b85bca652c3290138c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46371 } }
I20260925 04:39:27.103780 8637 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33070ba178534b85bca652c3290138c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:39:27.103870 8637 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33070ba178534b85bca652c3290138c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33070ba178534b85bca652c3290138c9, State: Initialized, Role: FOLLOWER
I20260925 04:39:27.104351 8637 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33070ba178534b85bca652c3290138c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33070ba178534b85bca652c3290138c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46371 } }
I20260925 04:39:27.104501 8637 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33070ba178534b85bca652c3290138c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:39:27.104565 8637 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33070ba178534b85bca652c3290138c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:39:27.104663 8637 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33070ba178534b85bca652c3290138c9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:39:27.105679 8637 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33070ba178534b85bca652c3290138c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33070ba178534b85bca652c3290138c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46371 } }
I20260925 04:39:27.106063 8637 leader_election.cc:302] T 00000000000000000000000000000000 P 33070ba178534b85bca652c3290138c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33070ba178534b85bca652c3290138c9; no voters:
I20260925 04:39:27.106403 8637 leader_election.cc:288] T 00000000000000000000000000000000 P 33070ba178534b85bca652c3290138c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:39:27.106530 8642 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33070ba178534b85bca652c3290138c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:39:27.106755 8642 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33070ba178534b85bca652c3290138c9 [term 1 LEADER]: Becoming Leader. State: Replica: 33070ba178534b85bca652c3290138c9, State: Running, Role: LEADER
I20260925 04:39:27.107086 8642 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33070ba178534b85bca652c3290138c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33070ba178534b85bca652c3290138c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46371 } }
I20260925 04:39:27.107813 8637 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33070ba178534b85bca652c3290138c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:39:27.108930 8644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33070ba178534b85bca652c3290138c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33070ba178534b85bca652c3290138c9. Latest consensus state: current_term: 1 leader_uuid: "33070ba178534b85bca652c3290138c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33070ba178534b85bca652c3290138c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46371 } } }
I20260925 04:39:27.108930 8643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33070ba178534b85bca652c3290138c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33070ba178534b85bca652c3290138c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33070ba178534b85bca652c3290138c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46371 } } }
I20260925 04:39:27.109099 8644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33070ba178534b85bca652c3290138c9 [sys.catalog]: This master's current role is: LEADER
I20260925 04:39:27.109099 8643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33070ba178534b85bca652c3290138c9 [sys.catalog]: This master's current role is: LEADER
I20260925 04:39:27.109534 8651 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:39:27.112922 8651 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:39:27.118988 8651 catalog_manager.cc:1347] Generated new cluster ID: 7ea8d7c8613c43fe90e0837bdbb48c54
I20260925 04:39:27.119079 8651 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:39:27.129778 8651 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:39:27.131135 8651 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:39:27.146104 8651 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33070ba178534b85bca652c3290138c9: Generated new TSK 0
I20260925 04:39:27.146901 8651 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 04:39:27.184952 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46371
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:39:27.298331 8663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:39:27.298633 8663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:39:27.298677 8663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:39:27.302922 8663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:39:27.303004 8663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:39:27.303074 8663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 04:39:27.308784 8663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8663
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:39:27.309947 8663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:39:27.310956 8663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:39:27.320902 8671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:27.320976 8663 server_base.cc:1034] running on GCE node
W20260925 04:39:27.320895 8668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:27.320895 8669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:27.321575 8663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:39:27.322264 8663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:39:27.323477 8663 hybrid_clock.cc:648] HybridClock initialized: now 1790311167323452 us; error 36 us; skew 500 ppm
Sep 25 04:39:27 dist-test-slave-dc95 krb5kdc[8556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311167, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:39:27.326304 8663 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 04:39:27.327225 8663 webserver.cc:460] Webserver started at http://127.0.98.1:40313/ using document root <none> and password file <none>
I20260925 04:39:27.327828 8663 fs_manager.cc:359] Metadata directory not provided
I20260925 04:39:27.327898 8663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:39:27.328095 8663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:39:27.330170 8663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-0/data/instance:
uuid: "6a66818e5e3549e992fd55b6f3aa92d8"
format_stamp: "Formatted at 2026-09-25 04:39:27 on dist-test-slave-dc95"
server_key: "21125b38c502355d69ad886044b0c9eb"
server_key_iv: "c8e2500505e9d7962b487dfc48df8efd"
server_key_version: "encryptionkey@0"
I20260925 04:39:27.330722 8663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6a66818e5e3549e992fd55b6f3aa92d8"
format_stamp: "Formatted at 2026-09-25 04:39:27 on dist-test-slave-dc95"
server_key: "21125b38c502355d69ad886044b0c9eb"
server_key_iv: "c8e2500505e9d7962b487dfc48df8efd"
server_key_version: "encryptionkey@0"
I20260925 04:39:27.334589 8663 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 04:39:27.337198 8678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:27.338531 8663 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 04:39:27.338668 8663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-0/wal
uuid: "6a66818e5e3549e992fd55b6f3aa92d8"
format_stamp: "Formatted at 2026-09-25 04:39:27 on dist-test-slave-dc95"
server_key: "21125b38c502355d69ad886044b0c9eb"
server_key_iv: "c8e2500505e9d7962b487dfc48df8efd"
server_key_version: "encryptionkey@0"
I20260925 04:39:27.338783 8663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:39:27.368444 8663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:39:27.371752 8663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:39:27.371944 8663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:39:27.372556 8663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:39:27.373405 8663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:39:27.373456 8663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:27.373509 8663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:39:27.373539 8663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:27.382503 8663 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46841
I20260925 04:39:27.382563 8791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46841 every 8 connection(s)
I20260925 04:39:27.383467 8663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
I20260925 04:39:27.391897 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 8663
I20260925 04:39:27.392035 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-0/wal/instance
I20260925 04:39:27.392352 392 external_mini_cluster.cc:1428] Setting key 0b387112ef281f774387a24a6e9ae3c1
Sep 25 04:39:27 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311167, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:39:27.398043 8640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:27.385467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43983 (local address 127.0.98.62:46371)
0925 04:39:27.385650 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:27.385655 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:27.386508 (+ 853us) server_negotiation.cc:407] Connection header received
0925 04:39:27.387870 (+ 1362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:27.387873 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:27.387930 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:27.388012 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:27.389292 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:27.389901 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:27.391075 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:27.391321 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:27.391406 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:27.394187 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:27.394191 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:27.394193 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:27.394217 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:27.395782 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:27.396287 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:27.396291 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:27.396293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:27.396389 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:27.396773 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:27.396776 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:27.396777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:27.396983 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:27.397060 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:27.397731 (+ 671us) server_negotiation.cc:299] Negotiation successful
0925 04:39:27.397899 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 04:39:27.399173 8794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:27.385869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46371 (local address 127.0.98.1:43983)
0925 04:39:27.386346 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:27.386395 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:27.387646 (+ 1251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:27.388229 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:27.388237 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:27.388619 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:27.389142 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:27.389158 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:27.390026 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:27.390029 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:27.390933 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:27.390943 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:27.391451 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:27.391454 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:27.391663 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:27.392359 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:27.392383 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:27.394014 (+ 1631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:39:27.395901 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:27.395907 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:27.395922 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:27.396193 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:27.396508 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:27.396511 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:27.396513 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:27.396685 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:27.397078 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:39:27.397084 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:39:27.397413 (+ 329us) client_negotiation.cc:769] Sending connection context
0925 04:39:27.397652 (+ 239us) client_negotiation.cc:240] Negotiation successful
0925 04:39:27.398044 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":119,"threads_started":1}
I20260925 04:39:27.400519 8792 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46371
I20260925 04:39:27.400794 8792 heartbeater.cc:464] Registering TS with master...
I20260925 04:39:27.401424 8792 heartbeater.cc:511] Master 127.0.98.62:46371 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 04:39:27.403064 8601 ts_manager.cc:194] Registered new tserver with Master: 6a66818e5e3549e992fd55b6f3aa92d8 (127.0.98.1:46841)
I20260925 04:39:27.404327 8601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43983
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 04:39:27.420836 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46371
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:39:27.534471 8797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:39:27.534793 8797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:39:27.534859 8797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:39:27.539093 8797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:39:27.539179 8797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:39:27.539265 8797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 04:39:27.545109 8797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:39:27.546285 8797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:39:27.547369 8797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:39:27.557765 8805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:27.557948 8802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:27.557879 8797 server_base.cc:1034] running on GCE node
W20260925 04:39:27.557801 8803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:27.558465 8797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:39:27.559125 8797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:39:27.560290 8797 hybrid_clock.cc:648] HybridClock initialized: now 1790311167560273 us; error 32 us; skew 500 ppm
Sep 25 04:39:27 dist-test-slave-dc95 krb5kdc[8556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311167, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:39:27.563102 8797 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 04:39:27.564093 8797 webserver.cc:460] Webserver started at http://127.0.98.2:37149/ using document root <none> and password file <none>
I20260925 04:39:27.564680 8797 fs_manager.cc:359] Metadata directory not provided
I20260925 04:39:27.564743 8797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:39:27.564934 8797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:39:27.567020 8797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-1/data/instance:
uuid: "16660e4f292343d48c4404faa199d5f1"
format_stamp: "Formatted at 2026-09-25 04:39:27 on dist-test-slave-dc95"
server_key: "2a59aeb14c3834951f89469b53e7a652"
server_key_iv: "716a26debd363ddc6726b9f77cd3ec6f"
server_key_version: "encryptionkey@0"
I20260925 04:39:27.567582 8797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-1/wal/instance:
uuid: "16660e4f292343d48c4404faa199d5f1"
format_stamp: "Formatted at 2026-09-25 04:39:27 on dist-test-slave-dc95"
server_key: "2a59aeb14c3834951f89469b53e7a652"
server_key_iv: "716a26debd363ddc6726b9f77cd3ec6f"
server_key_version: "encryptionkey@0"
I20260925 04:39:27.571380 8797 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 04:39:27.574121 8812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:27.575695 8797 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:39:27.575820 8797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-1/wal
uuid: "16660e4f292343d48c4404faa199d5f1"
format_stamp: "Formatted at 2026-09-25 04:39:27 on dist-test-slave-dc95"
server_key: "2a59aeb14c3834951f89469b53e7a652"
server_key_iv: "716a26debd363ddc6726b9f77cd3ec6f"
server_key_version: "encryptionkey@0"
I20260925 04:39:27.575951 8797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:39:27.588389 8797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:39:27.591840 8797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:39:27.592041 8797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:39:27.592662 8797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:39:27.593528 8797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:39:27.593577 8797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:27.593619 8797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:39:27.593648 8797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:27.602802 8797 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36053
I20260925 04:39:27.602898 8925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36053 every 8 connection(s)
I20260925 04:39:27.603852 8797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
I20260925 04:39:27.607307 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 8797
I20260925 04:39:27.607437 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-1/wal/instance
I20260925 04:39:27.607820 392 external_mini_cluster.cc:1428] Setting key 0073849b66121ebf35a36cb179cd8c78
Sep 25 04:39:27 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311167, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260925 04:39:27.620543 8640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:27.605767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57763 (local address 127.0.98.62:46371)
0925 04:39:27.605938 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:27.605943 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:27.606895 (+ 952us) server_negotiation.cc:407] Connection header received
0925 04:39:27.608707 (+ 1812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:27.608711 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:27.608764 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:27.608842 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:27.610554 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:27.611094 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:27.612840 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:27.613095 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:27.613191 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:27.616377 (+ 3186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:27.616381 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:27.616383 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:27.616414 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:27.617813 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:27.618531 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:27.618536 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:27.618538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:27.618644 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:27.619085 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:27.619088 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:27.619090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:27.619294 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:27.619374 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:27.620201 (+ 827us) server_negotiation.cc:299] Negotiation successful
0925 04:39:27.620406 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 04:39:27.621476 8928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:27.606232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46371 (local address 127.0.98.2:57763)
0925 04:39:27.606724 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:27.606772 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:27.608435 (+ 1663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:27.609106 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:27.609114 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:27.609656 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:27.610378 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:27.610399 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:27.611217 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:27.611221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:27.612689 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:27.612703 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:27.613270 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:27.613274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:27.613490 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:27.614150 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:27.614173 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:27.616136 (+ 1963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:39:27.617987 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:27.617994 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:27.618010 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:27.618313 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:27.618774 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:27.618777 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:27.618778 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:27.618955 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:27.619427 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:39:27.619435 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:39:27.619872 (+ 437us) client_negotiation.cc:769] Sending connection context
0925 04:39:27.620099 (+ 227us) client_negotiation.cc:240] Negotiation successful
0925 04:39:27.620448 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":130,"threads_started":1}
I20260925 04:39:27.622897 8926 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46371
I20260925 04:39:27.623194 8926 heartbeater.cc:464] Registering TS with master...
I20260925 04:39:27.623778 8926 heartbeater.cc:511] Master 127.0.98.62:46371 requested a full tablet report, sending...
I20260925 04:39:27.625113 8601 ts_manager.cc:194] Registered new tserver with Master: 16660e4f292343d48c4404faa199d5f1 (127.0.98.2:36053)
I20260925 04:39:27.625875 8601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57763
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 04:39:27.640204 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46371
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:39:27.755290 8931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:39:27.755570 8931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:39:27.755656 8931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:39:27.759902 8931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:39:27.759980 8931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:39:27.760066 8931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 04:39:27.765576 8931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:39:27.766685 8931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:39:27.767794 8931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:39:27.778335 8937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:27.778396 8939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:27.778369 8936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:27.778599 8931 server_base.cc:1034] running on GCE node
I20260925 04:39:27.779184 8931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:39:27.779920 8931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:39:27.781112 8931 hybrid_clock.cc:648] HybridClock initialized: now 1790311167781070 us; error 55 us; skew 500 ppm
Sep 25 04:39:27 dist-test-slave-dc95 krb5kdc[8556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311167, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:39:27.789966 8931 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 04:39:27.791026 8931 webserver.cc:460] Webserver started at http://127.0.98.3:45341/ using document root <none> and password file <none>
I20260925 04:39:27.791638 8931 fs_manager.cc:359] Metadata directory not provided
I20260925 04:39:27.791721 8931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:39:27.791941 8931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:39:27.794130 8931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-2/data/instance:
uuid: "f72bbbe9ffbf42768c798b32d8db38ca"
format_stamp: "Formatted at 2026-09-25 04:39:27 on dist-test-slave-dc95"
server_key: "5d978c39d75a30dfaf7a066edb1194af"
server_key_iv: "15d497f2ea0dc8d4f5873a2a9aa79411"
server_key_version: "encryptionkey@0"
I20260925 04:39:27.794714 8931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f72bbbe9ffbf42768c798b32d8db38ca"
format_stamp: "Formatted at 2026-09-25 04:39:27 on dist-test-slave-dc95"
server_key: "5d978c39d75a30dfaf7a066edb1194af"
server_key_iv: "15d497f2ea0dc8d4f5873a2a9aa79411"
server_key_version: "encryptionkey@0"
I20260925 04:39:27.798718 8931 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 04:39:27.801554 8946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:27.802855 8931 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:39:27.803010 8931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-2/wal
uuid: "f72bbbe9ffbf42768c798b32d8db38ca"
format_stamp: "Formatted at 2026-09-25 04:39:27 on dist-test-slave-dc95"
server_key: "5d978c39d75a30dfaf7a066edb1194af"
server_key_iv: "15d497f2ea0dc8d4f5873a2a9aa79411"
server_key_version: "encryptionkey@0"
I20260925 04:39:27.803124 8931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:39:27.825865 8931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:39:27.829242 8931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:39:27.829432 8931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:39:27.830068 8931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:39:27.830888 8931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:39:27.830935 8931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:27.830987 8931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:39:27.831017 8931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:27.839515 8931 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34659
I20260925 04:39:27.839550 9059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34659 every 8 connection(s)
I20260925 04:39:27.840412 8931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
I20260925 04:39:27.847400 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 8931
I20260925 04:39:27.847530 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-2/wal/instance
I20260925 04:39:27.847826 392 external_mini_cluster.cc:1428] Setting key 77bda613fd701af585502c44f13bbe85
Sep 25 04:39:27 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311167, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:39:27.853681 8640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:27.842143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46307 (local address 127.0.98.62:46371)
0925 04:39:27.842269 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:27.842273 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:27.843154 (+ 881us) server_negotiation.cc:407] Connection header received
0925 04:39:27.844632 (+ 1478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:27.844636 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:27.844695 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:27.844767 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:27.846013 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:27.846544 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:27.847727 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:27.847984 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:27.848076 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:27.850593 (+ 2517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:27.850596 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:27.850598 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:27.850623 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:27.851888 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:27.852389 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:27.852392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:27.852393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:27.852476 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:27.852771 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:27.852774 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:27.852775 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:27.852946 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:27.853027 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:27.853458 (+ 431us) server_negotiation.cc:299] Negotiation successful
0925 04:39:27.853560 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 04:39:27.854503 9062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:27.842540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46371 (local address 127.0.98.3:46307)
0925 04:39:27.843017 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:27.843060 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:27.844378 (+ 1318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:27.844958 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:27.844966 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:27.845364 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:27.845866 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:27.845880 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:27.846663 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:27.846665 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:27.847595 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:27.847632 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:27.848127 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:27.848131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:27.848328 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:27.848893 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:27.848910 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:27.850445 (+ 1535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:39:27.851996 (+ 1551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:27.852002 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:27.852014 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:27.852287 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:27.852575 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:27.852577 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:27.852578 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:27.852703 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:27.853025 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:39:27.853030 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:39:27.853278 (+ 248us) client_negotiation.cc:769] Sending connection context
0925 04:39:27.853429 (+ 151us) client_negotiation.cc:240] Negotiation successful
0925 04:39:27.853678 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":98,"threads_started":1}
I20260925 04:39:27.855566 9060 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46371
I20260925 04:39:27.855816 9060 heartbeater.cc:464] Registering TS with master...
I20260925 04:39:27.856343 9060 heartbeater.cc:511] Master 127.0.98.62:46371 requested a full tablet report, sending...
I20260925 04:39:27.857532 8601 ts_manager.cc:194] Registered new tserver with Master: f72bbbe9ffbf42768c798b32d8db38ca (127.0.98.3:34659)
I20260925 04:39:27.858335 8601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46307
I20260925 04:39:27.861900 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260925 04:39:27.938153 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 8663
I20260925 04:39:27.943116 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 8797
I20260925 04:39:27.948267 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 8931
I20260925 04:39:27.952991 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 8569
I20260925 04:39:27.958480 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46371
--webserver_interface=127.0.98.62
--webserver_port=37071
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46371
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:39:28.071466 9069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:39:28.071837 9069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:39:28.071887 9069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:39:28.076160 9069 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:39:28.076242 9069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:39:28.076261 9069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:39:28.076277 9069 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:39:28.076292 9069 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:39:28.082589 9069 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46371
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=37071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:39:28.083822 9069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:39:28.084868 9069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:39:28.093657 9075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:28.093678 9077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:28.093678 9074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:28.094040 9069 server_base.cc:1034] running on GCE node
I20260925 04:39:28.094661 9069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:39:28.095896 9069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:39:28.097097 9069 hybrid_clock.cc:648] HybridClock initialized: now 1790311168097073 us; error 39 us; skew 500 ppm
Sep 25 04:39:28 dist-test-slave-dc95 krb5kdc[8556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:39:28.099928 9069 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 04:39:28.100920 9069 webserver.cc:460] Webserver started at http://127.0.98.62:37071/ using document root <none> and password file <none>
I20260925 04:39:28.101485 9069 fs_manager.cc:359] Metadata directory not provided
I20260925 04:39:28.101564 9069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:39:28.106101 9069 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.004s
I20260925 04:39:28.108333 9084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:28.109663 9069 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:39:28.109783 9069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/master-0/wal
uuid: "33070ba178534b85bca652c3290138c9"
format_stamp: "Formatted at 2026-09-25 04:39:27 on dist-test-slave-dc95"
server_key: "e5a47f49f50d2f019843bfcce8459467"
server_key_iv: "db06888b9c35de5281e81750d1506fcf"
server_key_version: "encryptionkey@0"
I20260925 04:39:28.110359 9069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:39:28.123961 9069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:39:28.127368 9069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:39:28.127548 9069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:39:28.134404 9069 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46371
I20260925 04:39:28.134411 9136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46371 every 8 connection(s)
I20260925 04:39:28.135478 9069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
I20260925 04:39:28.139765 9137 sys_catalog.cc:263] Verifying existing consensus state
I20260925 04:39:28.141542 9137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33070ba178534b85bca652c3290138c9: Bootstrap starting.
I20260925 04:39:28.144894 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 9069
I20260925 04:39:28.145323 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:46841
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:46371
--webserver_port=40313
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:39:28.158802 9137 log.cc:824] T 00000000000000000000000000000000 P 33070ba178534b85bca652c3290138c9: Log is configured to *not* fsync() on all Append() calls
I20260925 04:39:28.166142 9137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33070ba178534b85bca652c3290138c9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 04:39:28.166503 9137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33070ba178534b85bca652c3290138c9: Bootstrap complete.
I20260925 04:39:28.170627 9137 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33070ba178534b85bca652c3290138c9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33070ba178534b85bca652c3290138c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46371 } }
I20260925 04:39:28.171073 9137 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33070ba178534b85bca652c3290138c9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33070ba178534b85bca652c3290138c9, State: Initialized, Role: FOLLOWER
I20260925 04:39:28.171564 9137 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33070ba178534b85bca652c3290138c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33070ba178534b85bca652c3290138c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46371 } }
I20260925 04:39:28.171736 9137 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33070ba178534b85bca652c3290138c9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:39:28.171803 9137 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33070ba178534b85bca652c3290138c9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:39:28.171901 9137 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33070ba178534b85bca652c3290138c9 [term 1 FOLLOWER]: Advancing to term 2
I20260925 04:39:28.173997 9137 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33070ba178534b85bca652c3290138c9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33070ba178534b85bca652c3290138c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46371 } }
I20260925 04:39:28.174350 9137 leader_election.cc:302] T 00000000000000000000000000000000 P 33070ba178534b85bca652c3290138c9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33070ba178534b85bca652c3290138c9; no voters:
I20260925 04:39:28.174700 9137 leader_election.cc:288] T 00000000000000000000000000000000 P 33070ba178534b85bca652c3290138c9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 04:39:28.174818 9141 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33070ba178534b85bca652c3290138c9 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 04:39:28.175014 9141 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33070ba178534b85bca652c3290138c9 [term 2 LEADER]: Becoming Leader. State: Replica: 33070ba178534b85bca652c3290138c9, State: Running, Role: LEADER
I20260925 04:39:28.175333 9141 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33070ba178534b85bca652c3290138c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33070ba178534b85bca652c3290138c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46371 } }
I20260925 04:39:28.176069 9137 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33070ba178534b85bca652c3290138c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:39:28.176769 9142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33070ba178534b85bca652c3290138c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "33070ba178534b85bca652c3290138c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33070ba178534b85bca652c3290138c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46371 } } }
I20260925 04:39:28.176932 9142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33070ba178534b85bca652c3290138c9 [sys.catalog]: This master's current role is: LEADER
I20260925 04:39:28.177263 9143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33070ba178534b85bca652c3290138c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33070ba178534b85bca652c3290138c9. Latest consensus state: current_term: 2 leader_uuid: "33070ba178534b85bca652c3290138c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33070ba178534b85bca652c3290138c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46371 } } }
I20260925 04:39:28.177404 9143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33070ba178534b85bca652c3290138c9 [sys.catalog]: This master's current role is: LEADER
I20260925 04:39:28.177692 9150 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:39:28.181331 9150 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:39:28.182130 9150 catalog_manager.cc:1259] Loaded cluster ID: 7ea8d7c8613c43fe90e0837bdbb48c54
I20260925 04:39:28.182201 9150 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:39:28.184679 9150 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:39:28.185494 9150 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 33070ba178534b85bca652c3290138c9: Loaded TSK: 0
I20260925 04:39:28.186817 9150 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 04:39:28.265425 9139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:39:28.265734 9139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:39:28.265785 9139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:39:28.270087 9139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:39:28.270174 9139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:39:28.270238 9139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 04:39:28.275943 9139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:46841
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=40313
--tserver_master_addrs=127.0.98.62:46371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:39:28.277119 9139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:39:28.278411 9139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:39:28.289345 9164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:28.289347 9163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:28.289774 9139 server_base.cc:1034] running on GCE node
W20260925 04:39:28.289597 9166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:28.290448 9139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:39:28.291162 9139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:39:28.292352 9139 hybrid_clock.cc:648] HybridClock initialized: now 1790311168292331 us; error 43 us; skew 500 ppm
Sep 25 04:39:28 dist-test-slave-dc95 krb5kdc[8556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311168, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:39:28.295430 9139 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260925 04:39:28.296491 9139 webserver.cc:460] Webserver started at http://127.0.98.1:40313/ using document root <none> and password file <none>
I20260925 04:39:28.297086 9139 fs_manager.cc:359] Metadata directory not provided
I20260925 04:39:28.297173 9139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:39:28.302105 9139 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260925 04:39:28.304792 9173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:28.306339 9139 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:39:28.306491 9139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-0/wal
uuid: "6a66818e5e3549e992fd55b6f3aa92d8"
format_stamp: "Formatted at 2026-09-25 04:39:27 on dist-test-slave-dc95"
server_key: "21125b38c502355d69ad886044b0c9eb"
server_key_iv: "c8e2500505e9d7962b487dfc48df8efd"
server_key_version: "encryptionkey@0"
I20260925 04:39:28.307106 9139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:39:28.343257 9139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:39:28.346499 9139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:39:28.346709 9139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:39:28.347321 9139 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:39:28.348205 9139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:39:28.348256 9139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:28.348309 9139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:39:28.348337 9139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:28.357184 9139 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46841
I20260925 04:39:28.357206 9286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46841 every 8 connection(s)
I20260925 04:39:28.358248 9139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
I20260925 04:39:28.363061 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 9139
I20260925 04:39:28.363481 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:36053
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:46371
--webserver_port=37149
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 04:39:28 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:39:28 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 04:39:28.373026 9289 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:39:28.360469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53215 (local address 127.0.98.62:46371)
0925 04:39:28.361015 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:28.361035 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:28.361258 (+ 223us) server_negotiation.cc:407] Connection header received
0925 04:39:28.362822 (+ 1564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:28.362834 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:28.363154 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:28.363503 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:28.364753 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:28.365747 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:28.367393 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:28.368025 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:28.368170 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:28.372283 (+ 4113us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53215: BlockingRecv error: recv got EOF from 127.0.98.1:53215 (error 108)
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":143,"threads_started":1}
I20260925 04:39:28.373126 9290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:28.360572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46371 (local address 127.0.98.1:53215)
0925 04:39:28.361092 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:28.361144 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:28.362392 (+ 1248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:28.363601 (+ 1209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:28.363636 (+ 35us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:28.364087 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:28.364573 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:28.364587 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:28.365902 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:28.365905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:28.367239 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:28.367253 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:28.368204 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:28.368210 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:28.368428 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:28.369052 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:28.369076 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:28.372027 (+ 2951us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46371: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":130,"threads_started":1}
W20260925 04:39:28.373746 9287 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46371 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:46371: Client connection negotiation failed: client connection to 127.0.98.62:46371: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 25 04:39:28 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:39:28 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:39:28.380894 9290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:28.374292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46371 (local address 127.0.98.1:36785)
0925 04:39:28.374500 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:28.374519 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:28.374618 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:28.374903 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:28.374906 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:28.375096 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:28.375215 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:28.375222 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:28.376242 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:28.376245 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:28.377376 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:28.377392 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:28.377974 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:28.377978 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:28.378116 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:28.378667 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:28.378682 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:28.380745 (+ 2063us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46371: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136}
W20260925 04:39:28.381028 9289 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:39:28.374376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36785 (local address 127.0.98.62:46371)
0925 04:39:28.374499 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:28.374503 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:28.374574 (+ 71us) server_negotiation.cc:407] Connection header received
0925 04:39:28.374742 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:28.374745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:28.374796 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:28.374877 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:28.375391 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:28.376111 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:28.377548 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:28.377854 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:28.377950 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:28.380916 (+ 2966us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:36785: BlockingRecv error: recv got EOF from 127.0.98.1:36785 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 25 04:39:28 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:39:28 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:39:28.386631 9290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:28.381633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46371 (local address 127.0.98.1:50903)
0925 04:39:28.381749 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:28.381763 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:28.381855 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:28.382095 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:28.382097 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:28.382253 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:28.382335 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:28.382341 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:28.383056 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:28.383060 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:28.383596 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:28.383644 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:28.384223 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:28.384227 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:28.384311 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:28.384787 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:28.384799 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:28.386492 (+ 1693us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46371: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 04:39:28.386765 9289 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:39:28.381719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50903 (local address 127.0.98.62:46371)
0925 04:39:28.381826 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:28.381830 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:28.381843 (+ 13us) server_negotiation.cc:407] Connection header received
0925 04:39:28.381949 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:28.381952 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:28.381996 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:28.382085 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:28.382442 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:28.382950 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:28.383753 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:28.384068 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:28.384201 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:28.386653 (+ 2452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50903: BlockingRecv error: recv got EOF from 127.0.98.1:50903 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260925 04:39:28.387070 9287 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 04:39:28.482580 9291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:39:28.482925 9291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:39:28.482975 9291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:39:28.488183 9291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:39:28.488271 9291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:39:28.488339 9291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 04:39:28.494462 9291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:36053
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37149
--tserver_master_addrs=127.0.98.62:46371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:39:28.495570 9291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:39:28.496675 9291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:39:28.506841 9297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:28.506861 9296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:28.507002 9291 server_base.cc:1034] running on GCE node
W20260925 04:39:28.506861 9299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:28.507782 9291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:39:28.508523 9291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:39:28.509716 9291 hybrid_clock.cc:648] HybridClock initialized: now 1790311168509689 us; error 43 us; skew 500 ppm
Sep 25 04:39:28 dist-test-slave-dc95 krb5kdc[8556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311168, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:39:28.512764 9291 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260925 04:39:28.513770 9291 webserver.cc:460] Webserver started at http://127.0.98.2:37149/ using document root <none> and password file <none>
I20260925 04:39:28.514362 9291 fs_manager.cc:359] Metadata directory not provided
I20260925 04:39:28.514464 9291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:39:28.518776 9291 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:39:28.521173 9306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:28.522594 9291 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 04:39:28.522753 9291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-1/wal
uuid: "16660e4f292343d48c4404faa199d5f1"
format_stamp: "Formatted at 2026-09-25 04:39:27 on dist-test-slave-dc95"
server_key: "2a59aeb14c3834951f89469b53e7a652"
server_key_iv: "716a26debd363ddc6726b9f77cd3ec6f"
server_key_version: "encryptionkey@0"
I20260925 04:39:28.523334 9291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:39:28.545253 9291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:39:28.548655 9291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:39:28.548848 9291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:39:28.549448 9291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:39:28.550323 9291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:39:28.550374 9291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:28.550429 9291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:39:28.550458 9291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:28.559165 9291 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36053
I20260925 04:39:28.559186 9419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36053 every 8 connection(s)
I20260925 04:39:28.560227 9291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
I20260925 04:39:28.561134 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 9291
I20260925 04:39:28.561625 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:34659
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:46371
--webserver_port=45341
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 04:39:28 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:39:28 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 04:39:28.572820 9289 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:39:28.562185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49115 (local address 127.0.98.62:46371)
0925 04:39:28.562309 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:28.562313 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:28.563514 (+ 1201us) server_negotiation.cc:407] Connection header received
0925 04:39:28.565133 (+ 1619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:28.565138 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:28.565203 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:28.565295 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:28.566603 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:28.567426 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:28.568674 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:28.569036 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:28.569158 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:28.572683 (+ 3525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49115: BlockingRecv error: recv got EOF from 127.0.98.2:49115 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 04:39:28.573379 9423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:28.562757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46371 (local address 127.0.98.2:49115)
0925 04:39:28.563314 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:28.563374 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:28.564869 (+ 1495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:28.565457 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:28.565464 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:28.565923 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:28.566431 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:28.566446 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:28.567569 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:28.567572 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:28.568534 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:28.568544 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:28.569190 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:28.569193 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:28.569346 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:28.569949 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:28.569965 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:28.572485 (+ 2520us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46371: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":148,"threads_started":1}
W20260925 04:39:28.573915 9420 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46371 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:46371: Client connection negotiation failed: client connection to 127.0.98.62:46371: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 25 04:39:28 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:39:28 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:39:28.579641 9423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:28.574389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46371 (local address 127.0.98.2:36147)
0925 04:39:28.574519 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:28.574533 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:28.574624 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:28.574863 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:28.574866 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:28.575023 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:28.575117 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:28.575123 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:28.575895 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:28.575899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:28.576508 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:28.576518 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:28.577019 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:28.577023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:28.577134 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:28.577608 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:28.577620 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:28.579476 (+ 1856us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46371: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 04:39:28.579771 9289 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:39:28.574505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36147 (local address 127.0.98.62:46371)
0925 04:39:28.574621 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:28.574625 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:28.574640 (+ 15us) server_negotiation.cc:407] Connection header received
0925 04:39:28.574729 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:28.574731 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:28.574770 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:28.574833 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:28.575235 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:28.575800 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:28.576621 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:28.576913 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:28.577003 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:28.579645 (+ 2642us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36147: BlockingRecv error: recv got EOF from 127.0.98.2:36147 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 25 04:39:28 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:39:28 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:39:28.585752 9423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:28.580328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46371 (local address 127.0.98.2:49389)
0925 04:39:28.580474 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:28.580490 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:28.580590 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:28.580812 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:28.580815 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:28.580999 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:28.581106 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:28.581113 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:28.581825 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:28.581829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:28.582622 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:28.582635 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:28.583042 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:28.583045 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:28.583126 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:28.583679 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:28.583692 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:28.585608 (+ 1916us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46371: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 04:39:28.585986 9289 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:39:28.580401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49389 (local address 127.0.98.62:46371)
0925 04:39:28.580500 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:28.580503 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:28.580538 (+ 35us) server_negotiation.cc:407] Connection header received
0925 04:39:28.580676 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:28.580678 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:28.580717 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:28.580779 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:28.581217 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:28.581720 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:28.582734 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:28.582938 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:28.583007 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:28.585783 (+ 2776us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49389: BlockingRecv error: recv got EOF from 127.0.98.2:49389 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260925 04:39:28.586169 9420 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 04:39:28.678254 9422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:39:28.678548 9422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:39:28.678601 9422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:39:28.682796 9422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:39:28.682871 9422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:39:28.682934 9422 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 04:39:28.688767 9422 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:34659
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=45341
--tserver_master_addrs=127.0.98.62:46371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9422
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:39:28.689827 9422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:39:28.690896 9422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:39:28.700920 9429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:28.700913 9431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:28.701012 9422 server_base.cc:1034] running on GCE node
W20260925 04:39:28.700913 9428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:28.701620 9422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:39:28.702375 9422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:39:28.703579 9422 hybrid_clock.cc:648] HybridClock initialized: now 1790311168703535 us; error 56 us; skew 500 ppm
Sep 25 04:39:28 dist-test-slave-dc95 krb5kdc[8556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311168, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:39:28.706498 9422 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260925 04:39:28.707460 9422 webserver.cc:460] Webserver started at http://127.0.98.3:45341/ using document root <none> and password file <none>
I20260925 04:39:28.708050 9422 fs_manager.cc:359] Metadata directory not provided
I20260925 04:39:28.708155 9422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:39:28.712195 9422 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 04:39:28.714325 9438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:28.715708 9422 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:39:28.715823 9422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-2/wal
uuid: "f72bbbe9ffbf42768c798b32d8db38ca"
format_stamp: "Formatted at 2026-09-25 04:39:27 on dist-test-slave-dc95"
server_key: "5d978c39d75a30dfaf7a066edb1194af"
server_key_iv: "15d497f2ea0dc8d4f5873a2a9aa79411"
server_key_version: "encryptionkey@0"
I20260925 04:39:28.716388 9422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:39:28.734385 9422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:39:28.737524 9422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:39:28.737720 9422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:39:28.738337 9422 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:39:28.739192 9422 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:39:28.739239 9422 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:28.739293 9422 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:39:28.739323 9422 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:28.748211 9422 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34659
I20260925 04:39:28.748240 9551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34659 every 8 connection(s)
I20260925 04:39:28.749253 9422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 04:39:28 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:39:28.758880 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 9422
Sep 25 04:39:28 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 04:39:28.760293 9289 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:39:28.751069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55461 (local address 127.0.98.62:46371)
0925 04:39:28.751199 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:28.751202 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:28.752088 (+ 886us) server_negotiation.cc:407] Connection header received
0925 04:39:28.753431 (+ 1343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:28.753435 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:28.753496 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:28.753577 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:28.754820 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:28.755337 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:28.756499 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:28.756721 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:28.756808 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:28.760185 (+ 3377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55461: BlockingRecv error: recv got EOF from 127.0.98.3:55461 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 04:39:28.760949 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:28.751455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46371 (local address 127.0.98.3:55461)
0925 04:39:28.751935 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:28.751977 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:28.753220 (+ 1243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:28.753774 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:28.753781 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:28.754154 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:28.754651 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:28.754666 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:28.755455 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:28.755458 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:28.756379 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:28.756388 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:28.756861 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:28.756865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:28.757024 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:28.757571 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:28.757587 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:28.760039 (+ 2452us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46371: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":147,"threads_started":1}
W20260925 04:39:28.761534 9552 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46371 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:46371: Client connection negotiation failed: client connection to 127.0.98.62:46371: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 25 04:39:28 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:39:28 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:39:28.766928 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:28.761972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46371 (local address 127.0.98.3:42863)
0925 04:39:28.762074 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:28.762087 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:28.762175 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:28.762396 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:28.762398 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:28.762526 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:28.762620 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:28.762624 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:28.763453 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:28.763456 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:28.764076 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:28.764085 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:28.764602 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:28.764605 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:28.764670 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:28.765108 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:28.765121 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:28.766818 (+ 1697us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46371: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260925 04:39:28.767094 9289 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:39:28.762052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42863 (local address 127.0.98.62:46371)
0925 04:39:28.762165 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:28.762168 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:28.762177 (+ 9us) server_negotiation.cc:407] Connection header received
0925 04:39:28.762255 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:28.762257 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:28.762300 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:28.762372 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:28.762713 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:28.763359 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:28.764171 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:28.764483 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:28.764608 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:28.766977 (+ 2369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:42863: BlockingRecv error: recv got EOF from 127.0.98.3:42863 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 04:39:28.769331 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:28.759949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41414 (local address 127.0.98.62:46371)
0925 04:39:28.760192 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:28.760196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:28.760210 (+ 14us) server_negotiation.cc:407] Connection header received
0925 04:39:28.760249 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:28.760252 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:28.760297 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:28.760385 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:28.761318 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:28.761985 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:28.763197 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:28.763436 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:28.763538 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:28.765091 (+ 1553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:28.765099 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:28.765123 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:28.765153 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:28.766971 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:28.767492 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:28.767499 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:28.767506 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:28.767719 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:28.768016 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:28.768020 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:28.768022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:28.768535 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:28.768699 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:28.768958 (+ 259us) server_negotiation.cc:299] Negotiation successful
0925 04:39:28.769174 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":66,"threads_started":1}
Sep 25 04:39:28 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:39:28 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:39:28.772665 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:28.767682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46371 (local address 127.0.98.3:55309)
0925 04:39:28.767852 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:28.767864 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:28.767928 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:28.768126 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:28.768128 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:28.768295 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:28.768384 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:28.768390 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:28.769236 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:28.769239 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:28.769965 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:28.769975 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:28.770398 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:28.770400 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:28.770459 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:28.770862 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:28.770874 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:28.772572 (+ 1698us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46371: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104}
W20260925 04:39:28.772771 9289 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:39:28.767733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55309 (local address 127.0.98.62:46371)
0925 04:39:28.767866 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:28.767870 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:28.767900 (+ 30us) server_negotiation.cc:407] Connection header received
0925 04:39:28.767986 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:28.767989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:28.768033 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:28.768093 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:28.768472 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:28.769126 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:28.770083 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:28.770299 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:28.770389 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:28.772680 (+ 2291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55309: BlockingRecv error: recv got EOF from 127.0.98.3:55309 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260925 04:39:28.772845 9552 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 25 04:39:29 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:39:29 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:39:29.393411 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:29.387959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46371 (local address 127.0.98.1:35777)
0925 04:39:29.388249 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:29.388263 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:29.388372 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:29.388620 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:29.388622 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:29.388791 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:29.388896 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:29.388901 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:29.389633 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:29.389636 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:29.390213 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:29.390221 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:29.390675 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:29.390678 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:29.390754 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:29.391240 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:29.391254 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:29.393286 (+ 2032us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46371: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":101,"threads_started":1}
W20260925 04:39:29.393590 9558 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:39:29.387994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35777 (local address 127.0.98.62:46371)
0925 04:39:29.388254 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:29.388258 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:29.388338 (+ 80us) server_negotiation.cc:407] Connection header received
0925 04:39:29.388454 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:29.388456 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:29.388507 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:29.388598 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:29.388994 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:29.389537 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:29.390329 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:29.390575 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:29.390674 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:29.393452 (+ 2778us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35777: BlockingRecv error: recv got EOF from 127.0.98.1:35777 (error 108)
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":89,"threads_started":1}
Sep 25 04:39:29 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:39:29 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:39:29.592649 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:29.586799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46371 (local address 127.0.98.2:54623)
0925 04:39:29.587075 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:29.587088 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:29.587192 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:29.587419 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:29.587421 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:29.587595 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:29.587791 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:29.587799 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:29.588557 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:29.588561 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:29.589156 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:29.589166 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:29.589712 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:29.589727 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:29.589833 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:29.590354 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:29.590365 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:29.592513 (+ 2148us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46371: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":97,"threads_started":1}
W20260925 04:39:29.592850 9558 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:39:29.586926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54623 (local address 127.0.98.62:46371)
0925 04:39:29.587078 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:29.587082 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:29.587159 (+ 77us) server_negotiation.cc:407] Connection header received
0925 04:39:29.587266 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:29.587268 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:29.587317 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:29.587388 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:29.587911 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:29.588434 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:29.589298 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:29.589520 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:29.589615 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:29.592729 (+ 3114us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54623: BlockingRecv error: recv got EOF from 127.0.98.2:54623 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 25 04:39:29 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:39:29 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:39:29.779376 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:29.773616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46371 (local address 127.0.98.3:51059)
0925 04:39:29.773940 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:29.773954 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:29.774059 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:29.774324 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:29.774327 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:29.774511 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:29.774640 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:29.774646 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:29.775491 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:29.775494 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:29.776168 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:29.776180 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:29.776759 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:29.776763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:29.776860 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:29.777422 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:29.777439 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:29.779240 (+ 1801us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46371: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":104,"threads_started":1}
W20260925 04:39:29.779546 9558 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:39:29.773781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51059 (local address 127.0.98.62:46371)
0925 04:39:29.773931 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:29.773935 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:29.774011 (+ 76us) server_negotiation.cc:407] Connection header received
0925 04:39:29.774139 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:29.774142 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:29.774188 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:29.774250 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:29.774793 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:29.775363 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:29.776312 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:29.776602 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:29.776712 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:29.779420 (+ 2708us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51059: BlockingRecv error: recv got EOF from 127.0.98.3:51059 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 25 04:39:30 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:39:30 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:39:30.399684 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:30.394186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46371 (local address 127.0.98.1:37647)
0925 04:39:30.394466 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:30.394486 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:30.394595 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:30.394851 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:30.394854 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:30.395054 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:30.395150 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:30.395156 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:30.395992 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:30.395996 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:30.396560 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:30.396568 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:30.397039 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:30.397043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:30.397126 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:30.397685 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:30.397698 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:30.399469 (+ 1771us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46371: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":83,"threads_started":1}
W20260925 04:39:30.399789 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:39:30.394238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37647 (local address 127.0.98.62:46371)
0925 04:39:30.394518 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:30.394522 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:30.394538 (+ 16us) server_negotiation.cc:407] Connection header received
0925 04:39:30.394694 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:30.394697 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:30.394745 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:30.394819 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:30.395278 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:30.395885 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:30.396668 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:30.396929 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:30.397017 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:30.399667 (+ 2650us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37647: BlockingRecv error: recv got EOF from 127.0.98.1:37647 (error 108)
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":79,"threads_started":1}
Sep 25 04:39:30 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:39:30 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:39:30.599128 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:30.593597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46371 (local address 127.0.98.2:49039)
0925 04:39:30.593937 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:30.593953 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:30.594085 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:30.594348 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:30.594351 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:30.594550 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:30.594642 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:30.594648 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:30.595410 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:30.595413 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:30.596024 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:30.596034 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:30.596464 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:30.596467 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:30.596544 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:30.597051 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:30.597062 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:30.598968 (+ 1906us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46371: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":97,"threads_started":1}
W20260925 04:39:30.599476 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:39:30.593737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49039 (local address 127.0.98.62:46371)
0925 04:39:30.593911 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:30.593915 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:30.594030 (+ 115us) server_negotiation.cc:407] Connection header received
0925 04:39:30.594179 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:30.594183 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:30.594232 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:30.594302 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:30.594766 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:30.595292 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:30.596138 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:30.596359 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:30.596447 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:30.599149 (+ 2702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49039: BlockingRecv error: recv got EOF from 127.0.98.2:49039 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 25 04:39:30 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:39:30 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:39:30.792060 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:30.780141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46371 (local address 127.0.98.3:56749)
0925 04:39:30.780432 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:30.780447 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:30.780542 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:30.780767 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:30.780769 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:30.780951 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:30.781051 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:30.781056 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:30.781892 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:30.781896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:30.782506 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:30.782515 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:30.783011 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:30.783014 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:30.783085 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:30.789421 (+ 6336us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:30.789436 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:30.791901 (+ 2465us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46371: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":86,"threads_started":1}
W20260925 04:39:30.792129 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:39:30.780337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56749 (local address 127.0.98.62:46371)
0925 04:39:30.780476 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:30.780479 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:30.780496 (+ 17us) server_negotiation.cc:407] Connection header received
0925 04:39:30.780618 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:30.780621 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:30.780665 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:30.780738 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:30.781190 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:30.781736 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:30.782654 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:30.782891 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:30.782990 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:30.792026 (+ 9036us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56749: BlockingRecv error: recv got EOF from 127.0.98.3:56749 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 25 04:39:31 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:39:31 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:39:31.406049 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:31.400655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46371 (local address 127.0.98.1:55361)
0925 04:39:31.400918 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:31.400933 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:31.401071 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:31.401345 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:31.401349 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:31.401552 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:31.401650 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:31.401656 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:31.402431 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:31.402434 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:31.403007 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:31.403015 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:31.403465 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:31.403468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:31.403538 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:31.404029 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:31.404040 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:31.405847 (+ 1807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46371: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":71,"threads_started":1}
W20260925 04:39:31.406121 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:39:31.400796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55361 (local address 127.0.98.62:46371)
0925 04:39:31.401069 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:31.401073 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:31.401089 (+ 16us) server_negotiation.cc:407] Connection header received
0925 04:39:31.401171 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:31.401174 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:31.401228 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:31.401317 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:31.401739 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:31.402334 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:31.403129 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:31.403364 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:31.403450 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:31.406006 (+ 2556us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55361: BlockingRecv error: recv got EOF from 127.0.98.1:55361 (error 108)
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":73,"threads_started":1}
Sep 25 04:39:31 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:39:31 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:39:31.605692 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:31.600148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46371 (local address 127.0.98.2:35503)
0925 04:39:31.600465 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:31.600479 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:31.600594 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:31.600857 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:31.600859 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:31.601074 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:31.601173 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:31.601179 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:31.601977 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:31.601980 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:31.602546 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:31.602555 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:31.603026 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:31.603030 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:31.603098 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:31.603700 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:31.603712 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:31.605561 (+ 1849us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46371: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":85,"threads_started":1}
W20260925 04:39:31.605846 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:39:31.600277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35503 (local address 127.0.98.62:46371)
0925 04:39:31.600435 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:31.600438 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:31.600540 (+ 102us) server_negotiation.cc:407] Connection header received
0925 04:39:31.600676 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:31.600679 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:31.600721 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:31.600784 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:31.601300 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:31.601837 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:31.602675 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:31.602898 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:31.603035 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:31.605722 (+ 2687us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35503: BlockingRecv error: recv got EOF from 127.0.98.2:35503 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 25 04:39:31 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:39:31 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:39:31.799152 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:31.793064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46371 (local address 127.0.98.3:55735)
0925 04:39:31.793454 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:31.793468 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:31.793603 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:31.793921 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:31.793924 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:31.794149 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:31.794274 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:31.794283 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:31.795227 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:31.795230 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:31.795903 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:31.795912 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:31.796403 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:31.796405 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:31.796470 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:31.797016 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:31.797031 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:31.798997 (+ 1966us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46371: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":81,"threads_started":1}
W20260925 04:39:31.799290 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:39:31.793282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55735 (local address 127.0.98.62:46371)
0925 04:39:31.793446 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:31.793450 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:31.793573 (+ 123us) server_negotiation.cc:407] Connection header received
0925 04:39:31.793693 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:31.793696 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:31.793758 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:31.793847 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:31.794410 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:31.795109 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:31.796064 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:31.796301 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:31.796403 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:31.799180 (+ 2777us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55735: BlockingRecv error: recv got EOF from 127.0.98.3:55735 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 25 04:39:32 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:39:32 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:39:32.412418 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:32.406883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46371 (local address 127.0.98.1:57263)
0925 04:39:32.407191 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:32.407205 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:32.407299 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:32.407517 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:32.407519 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:32.407745 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:32.407845 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:32.407851 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:32.408541 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:32.408544 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:32.409123 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:32.409130 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:32.409600 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:32.409604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:32.409675 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:32.410212 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:32.410224 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:32.412265 (+ 2041us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46371: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":88,"threads_started":1}
W20260925 04:39:32.412554 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:39:32.407003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57263 (local address 127.0.98.62:46371)
0925 04:39:32.407241 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:32.407244 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:32.407258 (+ 14us) server_negotiation.cc:407] Connection header received
0925 04:39:32.407380 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:32.407382 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:32.407424 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:32.407488 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:32.407946 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:32.408463 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:32.409222 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:32.409445 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:32.409523 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:32.412436 (+ 2913us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57263: BlockingRecv error: recv got EOF from 127.0.98.1:57263 (error 108)
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":94,"threads_started":1}
Sep 25 04:39:32 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:39:32 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:39:32.612195 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:32.606498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46371 (local address 127.0.98.2:38925)
0925 04:39:32.606805 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:32.606819 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:32.606920 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:32.607135 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:32.607137 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:32.607314 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:32.607407 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:32.607412 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:32.608185 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:32.608189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:32.608925 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:32.608938 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:32.609480 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:32.609484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:32.609573 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:32.610083 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:32.610097 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:32.611961 (+ 1864us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46371: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":90,"threads_started":1}
W20260925 04:39:32.612298 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:39:32.606584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38925 (local address 127.0.98.62:46371)
0925 04:39:32.606730 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:32.606734 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:32.606881 (+ 147us) server_negotiation.cc:407] Connection header received
0925 04:39:32.606990 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:32.606992 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:32.607036 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:32.607109 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:32.607494 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:32.608069 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:32.609113 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:32.609363 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:32.609462 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:32.612182 (+ 2720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:38925: BlockingRecv error: recv got EOF from 127.0.98.2:38925 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 25 04:39:32 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:39:32 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:39:32.809204 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:32.800120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46371 (local address 127.0.98.3:41347)
0925 04:39:32.800480 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:32.800493 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:32.800605 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:32.800855 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:32.800858 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:32.801068 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:32.801173 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:32.801179 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:32.801951 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:32.801954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:32.802523 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:32.802532 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:32.803019 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:32.803022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:32.803089 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:32.807222 (+ 4133us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:32.807238 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:32.809059 (+ 1821us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46371: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":93,"threads_started":1}
W20260925 04:39:32.809415 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:39:32.800220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41347 (local address 127.0.98.62:46371)
0925 04:39:32.800369 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:32.800374 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:32.800552 (+ 178us) server_negotiation.cc:407] Connection header received
0925 04:39:32.800697 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:32.800699 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:32.800746 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:32.800814 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:32.801292 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:32.801857 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:32.802654 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:32.802884 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:32.802974 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:32.809280 (+ 6306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41347: BlockingRecv error: recv got EOF from 127.0.98.3:41347 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 25 04:39:33 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:39:33 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:39:33.420557 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:33.413474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46371 (local address 127.0.98.1:49043)
0925 04:39:33.413860 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:33.413878 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:33.414007 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:33.414314 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:33.414318 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:33.414510 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:33.414628 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:33.414636 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:33.415889 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:33.415892 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:33.416656 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:33.416670 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:33.417401 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:33.417405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:33.417500 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:33.418086 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:33.418100 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:33.420415 (+ 2315us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46371: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":155,"threads_started":1}
W20260925 04:39:33.421046 9574 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:39:33.413590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49043 (local address 127.0.98.62:46371)
0925 04:39:33.413998 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:33.414002 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:33.414022 (+ 20us) server_negotiation.cc:407] Connection header received
0925 04:39:33.414084 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:33.414087 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:33.414140 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:33.414255 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:33.414797 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:33.415747 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:33.416796 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:33.417173 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:33.417275 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:33.420880 (+ 3605us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49043: BlockingRecv error: recv got EOF from 127.0.98.1:49043 (error 108)
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":98,"threads_started":1}
Sep 25 04:39:33 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:39:33 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:39:33.618839 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:33.612996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46371 (local address 127.0.98.2:50561)
0925 04:39:33.613483 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:33.613496 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:33.613609 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:33.613852 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:33.613855 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:33.614104 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:33.614216 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:33.614222 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:33.615014 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:33.615017 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:33.615728 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:33.615739 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:33.616218 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:33.616223 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:33.616289 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:33.616791 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:33.616804 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:33.618699 (+ 1895us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46371: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":101,"threads_started":1}
W20260925 04:39:33.618948 9574 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:39:33.613199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50561 (local address 127.0.98.62:46371)
0925 04:39:33.613368 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:33.613372 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:33.613572 (+ 200us) server_negotiation.cc:407] Connection header received
0925 04:39:33.613691 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:33.613694 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:33.613744 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:33.613819 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:33.614352 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:33.614915 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:33.615855 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:33.616080 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:33.616228 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:33.618847 (+ 2619us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50561: BlockingRecv error: recv got EOF from 127.0.98.2:50561 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 25 04:39:33 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:39:33 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:39:33.816020 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:33.810083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46371 (local address 127.0.98.3:36523)
0925 04:39:33.810413 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:33.810427 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:33.810537 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:33.810771 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:33.810774 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:33.810972 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:33.811092 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:33.811098 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:33.811943 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:33.811946 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:33.812579 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:33.812589 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:33.813070 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:33.813073 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:33.813148 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:33.813642 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:33.813655 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:33.815874 (+ 2219us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46371: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":83,"threads_started":1}
W20260925 04:39:33.816305 9574 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:39:33.810258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36523 (local address 127.0.98.62:46371)
0925 04:39:33.810411 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:33.810414 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:33.810529 (+ 115us) server_negotiation.cc:407] Connection header received
0925 04:39:33.810606 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:33.810608 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:33.810656 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:33.810725 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:33.811198 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:33.811818 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:33.812700 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:33.812932 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:33.813015 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:33.816179 (+ 3164us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36523: BlockingRecv error: recv got EOF from 127.0.98.3:36523 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 25 04:39:34 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:39:34 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:39:34.427856 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:34.421699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46371 (local address 127.0.98.1:33653)
0925 04:39:34.421987 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:34.422008 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:34.422112 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:34.422499 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:34.422501 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:34.422681 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:34.422808 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:34.422815 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:34.423678 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:34.423682 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:34.424277 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:34.424287 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:34.424772 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:34.424775 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:34.424851 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:34.425419 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:34.425431 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:34.427702 (+ 2271us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46371: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":85,"threads_started":1}
W20260925 04:39:34.428095 9578 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:39:34.421812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33653 (local address 127.0.98.62:46371)
0925 04:39:34.422057 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:34.422060 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:34.422076 (+ 16us) server_negotiation.cc:407] Connection header received
0925 04:39:34.422251 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:34.422254 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:34.422304 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:34.422429 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:34.422983 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:34.423524 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:34.424390 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:34.424625 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:34.424711 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:34.427854 (+ 3143us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33653: BlockingRecv error: recv got EOF from 127.0.98.1:33653 (error 108)
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":60,"threads_started":1}
Sep 25 04:39:34 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:39:34 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:39:34.624969 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:34.619649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46371 (local address 127.0.98.2:58351)
0925 04:39:34.619901 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:34.619914 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:34.620014 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:34.620229 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:34.620231 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:34.620434 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:34.620533 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:34.620540 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:34.621315 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:34.621319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:34.621907 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:34.621918 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:34.622436 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:34.622439 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:34.622514 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:34.623048 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:34.623060 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:34.624831 (+ 1771us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46371: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":67,"threads_started":1}
W20260925 04:39:34.625151 9578 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:39:34.619694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58351 (local address 127.0.98.62:46371)
0925 04:39:34.619819 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:34.619822 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:34.620043 (+ 221us) server_negotiation.cc:407] Connection header received
0925 04:39:34.620086 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:34.620089 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:34.620135 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:34.620208 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:34.620640 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:34.621179 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:34.622033 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:34.622280 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:34.622380 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:34.625022 (+ 2642us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58351: BlockingRecv error: recv got EOF from 127.0.98.2:58351 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 25 04:39:34 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:39:34 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:39:34.822643 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:34.816901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46371 (local address 127.0.98.3:53965)
0925 04:39:34.817183 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:34.817196 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:34.817309 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:34.817573 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:34.817575 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:34.817744 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:34.817839 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:34.817845 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:34.818565 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:34.818569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:34.819178 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:34.819188 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:34.819689 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:34.819692 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:34.819784 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:34.820462 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:34.820476 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:34.822490 (+ 2014us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46371: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":75,"threads_started":1}
W20260925 04:39:34.822742 9578 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:39:34.817026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53965 (local address 127.0.98.62:46371)
0925 04:39:34.817175 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:34.817179 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:34.817267 (+ 88us) server_negotiation.cc:407] Connection header received
0925 04:39:34.817414 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:34.817416 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:34.817464 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:34.817534 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:34.817930 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:34.818442 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:34.819293 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:34.819527 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:34.819631 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:34.822643 (+ 3012us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:53965: BlockingRecv error: recv got EOF from 127.0.98.3:53965 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 25 04:39:35 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:39:35 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:39:35.434448 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:35.428806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46371 (local address 127.0.98.1:56461)
0925 04:39:35.429207 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:35.429222 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:35.429344 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:35.429664 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:35.429668 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:35.429865 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:35.429963 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:35.429969 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:35.430753 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:35.430756 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:35.431321 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:35.431329 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:35.431854 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:35.431858 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:35.431943 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:35.432482 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:35.432498 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:35.434328 (+ 1830us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46371: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":126,"threads_started":1}
W20260925 04:39:35.434634 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:39:35.428966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56461 (local address 127.0.98.62:46371)
0925 04:39:35.429224 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:35.429228 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:35.429296 (+ 68us) server_negotiation.cc:407] Connection header received
0925 04:39:35.429479 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:35.429482 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:35.429537 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:35.429614 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:35.430088 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:35.430639 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:35.431444 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:35.431732 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:35.431831 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:35.434494 (+ 2663us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56461: BlockingRecv error: recv got EOF from 127.0.98.1:56461 (error 108)
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":80,"threads_started":1}
Sep 25 04:39:35 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:39:35 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:39:35.632237 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:35.625726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46371 (local address 127.0.98.2:44317)
0925 04:39:35.626172 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:35.626187 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:35.626284 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:35.626551 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:35.626554 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:35.626733 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:35.626835 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:35.626841 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:35.627735 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:35.627739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:35.628369 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:35.628380 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:35.628825 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:35.628829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:35.628913 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:35.629591 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:35.629604 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:35.632105 (+ 2501us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46371: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":83,"threads_started":1}
W20260925 04:39:35.632454 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:39:35.625779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44317 (local address 127.0.98.62:46371)
0925 04:39:35.625914 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:35.625918 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:35.626250 (+ 332us) server_negotiation.cc:407] Connection header received
0925 04:39:35.626354 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:35.626356 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:35.626406 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:35.626474 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:35.626967 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:35.627591 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:35.628496 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:35.628716 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:35.628805 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:35.632317 (+ 3512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44317: BlockingRecv error: recv got EOF from 127.0.98.2:44317 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 25 04:39:35 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:39:35 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:39:35.829546 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:35.823635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46371 (local address 127.0.98.3:39373)
0925 04:39:35.823943 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:35.823958 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:35.824077 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:35.824389 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:35.824391 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:35.824580 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:35.824681 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:35.824687 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:35.825472 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:35.825475 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:35.826047 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:35.826056 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:35.826500 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:35.826503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:35.826565 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:35.827169 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:35.827182 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:35.829378 (+ 2196us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46371: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":73,"threads_started":1}
W20260925 04:39:35.829799 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:39:35.823727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39373 (local address 127.0.98.62:46371)
0925 04:39:35.823878 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:35.823883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:35.824061 (+ 178us) server_negotiation.cc:407] Connection header received
0925 04:39:35.824218 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:35.824221 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:35.824278 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:35.824360 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:35.824793 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:35.825361 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:35.826173 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:35.826406 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:35.826485 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:35.829584 (+ 3099us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:39373: BlockingRecv error: recv got EOF from 127.0.98.3:39373 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 25 04:39:36 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:39:36 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:39:36.440626 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:36.435252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46371 (local address 127.0.98.1:36517)
0925 04:39:36.435549 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:36.435562 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:36.435730 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:36.435970 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:36.435972 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:36.436161 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:36.436259 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:36.436264 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:36.437004 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:36.437007 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:36.437574 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:36.437582 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:36.438025 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:36.438028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:36.438096 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:36.438628 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:36.438672 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:36.440497 (+ 1825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46371: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":81,"threads_started":1}
W20260925 04:39:36.440904 9586 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:39:36.435448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36517 (local address 127.0.98.62:46371)
0925 04:39:36.435723 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:36.435727 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:36.435740 (+ 13us) server_negotiation.cc:407] Connection header received
0925 04:39:36.435808 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:36.435810 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:36.435852 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:36.435928 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:36.436359 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:36.436902 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:36.437682 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:36.437912 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:36.437988 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:36.440770 (+ 2782us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:36517: BlockingRecv error: recv got EOF from 127.0.98.1:36517 (error 108)
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":68,"threads_started":1}
Sep 25 04:39:36 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:39:36 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:39:36.638968 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:36.633214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46371 (local address 127.0.98.2:60801)
0925 04:39:36.633502 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:36.633518 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:36.633654 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:36.633934 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:36.633937 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:36.634184 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:36.634305 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:36.634314 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:36.635344 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:36.635347 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:36.636028 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:36.636040 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:36.636548 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:36.636551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:36.636620 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:36.637090 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:36.637101 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:36.638848 (+ 1747us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46371: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":84,"threads_started":1}
W20260925 04:39:36.639109 9586 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:39:36.633339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60801 (local address 127.0.98.62:46371)
0925 04:39:36.633540 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:36.633544 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:36.633612 (+ 68us) server_negotiation.cc:407] Connection header received
0925 04:39:36.633748 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:36.633751 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:36.633802 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:36.633886 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:36.634454 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:36.635216 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:36.636183 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:36.636424 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:36.636513 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:36.639006 (+ 2493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60801: BlockingRecv error: recv got EOF from 127.0.98.2:60801 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
Sep 25 04:39:36 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:39:36 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:39:36.836002 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:36.830414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46371 (local address 127.0.98.3:48343)
0925 04:39:36.830663 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:36.830676 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:36.830820 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:36.831059 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:36.831061 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:36.831245 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:36.831342 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:36.831347 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:36.832162 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:36.832165 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:36.832787 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:36.832799 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:36.833384 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:36.833387 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:36.833459 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:36.833972 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:36.833984 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:36.835879 (+ 1895us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46371: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":71,"threads_started":1}
W20260925 04:39:36.836223 9586 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:39:36.830555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48343 (local address 127.0.98.62:46371)
0925 04:39:36.830680 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:36.830684 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:36.830756 (+ 72us) server_negotiation.cc:407] Connection header received
0925 04:39:36.830907 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:36.830909 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:36.830957 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:36.831033 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:36.831450 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:36.832029 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:36.832955 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:36.833246 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:36.833340 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:36.836062 (+ 2722us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:48343: BlockingRecv error: recv got EOF from 127.0.98.3:48343 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 25 04:39:37 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:39:37 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:39:37.447234 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:37.441501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46371 (local address 127.0.98.1:57957)
0925 04:39:37.441826 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:37.441840 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:37.441942 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:37.442189 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:37.442191 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:37.442367 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:37.442461 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:37.442467 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:37.443318 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:37.443322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:37.443954 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:37.443964 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:37.444469 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:37.444472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:37.444538 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:37.444997 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:37.445008 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:37.447085 (+ 2077us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46371: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":92,"threads_started":1}
W20260925 04:39:37.447315 9590 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:39:37.441645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57957 (local address 127.0.98.62:46371)
0925 04:39:37.441898 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:37.441902 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:37.441917 (+ 15us) server_negotiation.cc:407] Connection header received
0925 04:39:37.442031 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:37.442034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:37.442076 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:37.442155 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:37.442596 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:37.443147 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:37.444048 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:37.444362 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:37.444440 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:37.447209 (+ 2769us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57957: BlockingRecv error: recv got EOF from 127.0.98.1:57957 (error 108)
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":72,"threads_started":1}
Sep 25 04:39:37 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:39:37 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:39:37.645483 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:37.639846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46371 (local address 127.0.98.2:43403)
0925 04:39:37.640121 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:37.640134 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:37.640234 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:37.640545 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:37.640548 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:37.640734 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:37.640830 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:37.640836 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:37.641639 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:37.641643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:37.642279 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:37.642290 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:37.642763 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:37.642766 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:37.642835 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:37.643325 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:37.643337 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:37.645340 (+ 2003us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46371: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":79,"threads_started":1}
W20260925 04:39:37.645612 9590 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:39:37.639979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43403 (local address 127.0.98.62:46371)
0925 04:39:37.640137 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:37.640140 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:37.640197 (+ 57us) server_negotiation.cc:407] Connection header received
0925 04:39:37.640366 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:37.640369 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:37.640420 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:37.640494 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:37.640961 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:37.641507 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:37.642410 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:37.642635 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:37.642716 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:37.645509 (+ 2793us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:43403: BlockingRecv error: recv got EOF from 127.0.98.2:43403 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 25 04:39:37 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:39:37 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:39:37.848069 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:37.837006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46371 (local address 127.0.98.3:45225)
0925 04:39:37.837285 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:37.837301 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:37.837410 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:37.837695 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:37.837698 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:37.837906 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:37.838007 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:37.838014 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:37.838827 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:37.838831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:37.839420 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:37.839429 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:37.839915 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:37.839919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:37.839993 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:37.845983 (+ 5990us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:37.845998 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:37.847943 (+ 1945us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46371: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":76,"threads_started":1}
W20260925 04:39:37.848248 9590 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:39:37.837135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45225 (local address 127.0.98.62:46371)
0925 04:39:37.837297 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:37.837300 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:37.837401 (+ 101us) server_negotiation.cc:407] Connection header received
0925 04:39:37.837501 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:37.837504 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:37.837554 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:37.837619 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:37.838123 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:37.838694 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:37.839529 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:37.839777 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:37.839860 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:37.848123 (+ 8263us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45225: BlockingRecv error: recv got EOF from 127.0.98.3:45225 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 25 04:39:38 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:39:38 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:39:38.453344 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:38.448033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46371 (local address 127.0.98.1:57671)
0925 04:39:38.448294 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:38.448307 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:38.448400 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:38.448634 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:38.448636 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:38.448810 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:38.448951 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:38.448957 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:38.449754 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:38.449758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:38.450370 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:38.450380 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:38.450822 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:38.450825 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:38.450906 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:38.451403 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:38.451422 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:38.453221 (+ 1799us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46371: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":84,"threads_started":1}
W20260925 04:39:38.453529 9594 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:39:38.448128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57671 (local address 127.0.98.62:46371)
0925 04:39:38.448377 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:38.448380 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:38.448394 (+ 14us) server_negotiation.cc:407] Connection header received
0925 04:39:38.448480 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:38.448482 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:38.448523 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:38.448596 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:38.449082 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:38.449643 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:38.450492 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:38.450719 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:38.450798 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:38.453368 (+ 2570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57671: BlockingRecv error: recv got EOF from 127.0.98.1:57671 (error 108)
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":92,"threads_started":1}
Sep 25 04:39:38 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:39:38 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:39:38.651489 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:38.646295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46371 (local address 127.0.98.2:53117)
0925 04:39:38.646562 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:38.646577 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:38.646683 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:38.646895 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:38.646897 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:38.647070 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:38.647165 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:38.647170 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:38.647957 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:38.647961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:38.648560 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:38.648570 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:38.649040 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:38.649049 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:38.649120 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:38.649636 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:38.649647 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:38.651370 (+ 1723us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46371: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":89,"threads_started":1}
W20260925 04:39:38.651671 9594 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:39:38.646401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53117 (local address 127.0.98.62:46371)
0925 04:39:38.646542 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:38.646546 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:38.646648 (+ 102us) server_negotiation.cc:407] Connection header received
0925 04:39:38.646755 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:38.646757 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:38.646801 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:38.646866 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:38.647272 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:38.647834 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:38.648678 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:38.648910 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:38.648993 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:38.651519 (+ 2526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53117: BlockingRecv error: recv got EOF from 127.0.98.2:53117 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 25 04:39:38 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:39:38 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:39:38.854583 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:38.848952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46371 (local address 127.0.98.3:60583)
0925 04:39:38.849271 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:38.849288 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:38.849406 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:38.849689 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:38.849692 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:38.849883 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:38.849990 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:38.849998 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:38.850969 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:38.850972 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:38.851571 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:38.851579 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:38.852049 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:38.852053 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:38.852133 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:38.852652 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:38.852667 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:38.854428 (+ 1761us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46371: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":97,"threads_started":1}
W20260925 04:39:38.854705 9594 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:39:38.849169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60583 (local address 127.0.98.62:46371)
0925 04:39:38.849287 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:38.849290 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:38.849369 (+ 79us) server_negotiation.cc:407] Connection header received
0925 04:39:38.849515 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:38.849519 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:38.849571 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:38.849646 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:38.850154 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:38.850834 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:38.851719 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:38.851944 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:38.852031 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:38.854604 (+ 2573us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60583: BlockingRecv error: recv got EOF from 127.0.98.3:60583 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 25 04:39:39 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:39:39 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:39:39.459508 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:39.454190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46371 (local address 127.0.98.1:45231)
0925 04:39:39.454477 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:39.454491 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:39.454586 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:39.454819 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:39.454821 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:39.454997 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:39.455097 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:39.455102 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:39.455902 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:39.455906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:39.456505 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:39.456516 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:39.456969 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:39.456973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:39.457048 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:39.457538 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:39.457551 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:39.459374 (+ 1823us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46371: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":91,"threads_started":1}
W20260925 04:39:39.459718 9598 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:39:39.454294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45231 (local address 127.0.98.62:46371)
0925 04:39:39.454554 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:39.454557 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:39.454572 (+ 15us) server_negotiation.cc:407] Connection header received
0925 04:39:39.454666 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:39.454668 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:39.454712 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:39.454794 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:39.455194 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:39.455769 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:39.456610 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:39.456835 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:39.456910 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:39.459545 (+ 2635us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45231: BlockingRecv error: recv got EOF from 127.0.98.1:45231 (error 108)
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":71,"threads_started":1}
Sep 25 04:39:39 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:39:39 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:39:39.657829 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:39.652357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46371 (local address 127.0.98.2:57397)
0925 04:39:39.652670 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:39.652682 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:39.652786 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:39.653083 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:39.653086 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:39.653309 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:39.653409 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:39.653415 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:39.654227 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:39.654230 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:39.654813 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:39.654824 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:39.655302 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:39.655304 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:39.655362 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:39.655916 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:39.655928 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:39.657709 (+ 1781us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46371: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":99,"threads_started":1}
W20260925 04:39:39.658007 9598 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:39:39.652505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57397 (local address 127.0.98.62:46371)
0925 04:39:39.652643 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:39.652647 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:39.652800 (+ 153us) server_negotiation.cc:407] Connection header received
0925 04:39:39.652852 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:39.652855 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:39.652919 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:39.653003 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:39.653545 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:39.654076 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:39.654941 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:39.655206 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:39.655291 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:39.657885 (+ 2594us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:57397: BlockingRecv error: recv got EOF from 127.0.98.2:57397 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 25 04:39:39 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:39:39 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:39:39.862363 9600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:39.855450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46371 (local address 127.0.98.3:48027)
0925 04:39:39.855783 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:39.855797 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:39.855918 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:39.856224 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:39.856227 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:39.856443 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:39.856560 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:39.856569 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:39.857752 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:39.857756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:39.858690 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:39.858701 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:39.859320 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:39.859323 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:39.859407 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:39.860018 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:39.860033 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:39.862207 (+ 2174us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46371: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":151,"threads_started":1}
W20260925 04:39:39.862600 9598 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:39:39.855586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48027 (local address 127.0.98.62:46371)
0925 04:39:39.855759 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:39.855764 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:39.855880 (+ 116us) server_negotiation.cc:407] Connection header received
0925 04:39:39.856047 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:39.856057 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:39.856111 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:39.856180 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:39.856705 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:39.857602 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:39.858835 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:39.859208 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:39.859319 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:39.862362 (+ 3043us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:48027: BlockingRecv error: recv got EOF from 127.0.98.3:48027 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 25 04:39:40 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:39:40 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:39:40.465857 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:40.460454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46371 (local address 127.0.98.1:47055)
0925 04:39:40.460762 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:40.460776 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:40.460875 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:40.461116 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:40.461118 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:40.461303 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:40.461400 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:40.461405 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:40.462174 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:40.462177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:40.462762 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:40.462771 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:40.463189 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:40.463192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:40.463289 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:40.463860 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:40.463874 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:40.465736 (+ 1862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46371: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":100,"threads_started":1}
W20260925 04:39:40.466048 9602 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:39:40.460608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47055 (local address 127.0.98.62:46371)
0925 04:39:40.460822 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:40.460825 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:40.460839 (+ 14us) server_negotiation.cc:407] Connection header received
0925 04:39:40.460958 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:40.460961 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:40.461004 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:40.461083 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:40.461492 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:40.462052 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:40.462862 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:40.463095 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:40.463168 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:40.465907 (+ 2739us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47055: BlockingRecv error: recv got EOF from 127.0.98.1:47055 (error 108)
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":70,"threads_started":1}
Sep 25 04:39:40 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:39:40 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:39:40.664744 9603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:40.658843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46371 (local address 127.0.98.2:45025)
0925 04:39:40.659200 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:40.659216 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:40.659344 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:40.659600 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:40.659603 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:40.659830 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:40.659926 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:40.659932 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:40.660701 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:40.660705 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:40.661263 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:40.661272 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:40.661724 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:40.661728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:40.661814 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:40.662407 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:40.662420 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:40.664614 (+ 2194us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46371: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":111,"threads_started":1}
W20260925 04:39:40.665042 9602 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:39:40.658988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45025 (local address 127.0.98.62:46371)
0925 04:39:40.659202 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:40.659206 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:40.659296 (+ 90us) server_negotiation.cc:407] Connection header received
0925 04:39:40.659436 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:40.659439 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:40.659485 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:40.659553 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:40.660054 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:40.660586 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:40.661380 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:40.661606 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:40.661699 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:40.664845 (+ 3146us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45025: BlockingRecv error: recv got EOF from 127.0.98.2:45025 (error 108)
Metrics: {"server-negotiator.queue_time_us":126}
Sep 25 04:39:40 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:39:40 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:39:40.869695 9604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:40.863817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46371 (local address 127.0.98.3:54107)
0925 04:39:40.864134 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:40.864159 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:40.864290 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:40.864545 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:40.864548 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:40.864767 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:40.864865 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:40.864871 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:40.865645 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:40.865648 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:40.866210 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:40.866218 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:40.866673 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:40.866677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:40.866836 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:40.867385 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:40.867397 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:40.869546 (+ 2149us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46371: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":84,"threads_started":1}
W20260925 04:39:40.869808 9602 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:39:40.863968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54107 (local address 127.0.98.62:46371)
0925 04:39:40.864186 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:40.864191 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:40.864230 (+ 39us) server_negotiation.cc:407] Connection header received
0925 04:39:40.864389 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:40.864392 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:40.864438 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:40.864504 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:40.864988 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:40.865514 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:40.866329 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:40.866552 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:40.866642 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:40.869689 (+ 3047us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54107: BlockingRecv error: recv got EOF from 127.0.98.3:54107 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 25 04:39:41 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:39:41 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:39:41.472225 9605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:41.466667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46371 (local address 127.0.98.1:57475)
0925 04:39:41.466950 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:41.466963 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:41.467053 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:41.467307 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:41.467309 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:41.467485 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:41.467581 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:41.467586 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:41.468474 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:41.468478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:41.469058 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:41.469067 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:41.469557 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:41.469559 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:41.469622 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:41.470149 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:41.470162 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:41.472096 (+ 1934us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46371: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":80,"threads_started":1}
W20260925 04:39:41.472479 9606 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:39:41.466762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57475 (local address 127.0.98.62:46371)
0925 04:39:41.467010 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:41.467014 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:41.467028 (+ 14us) server_negotiation.cc:407] Connection header received
0925 04:39:41.467136 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:41.467139 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:41.467182 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:41.467262 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:41.467757 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:41.468324 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:41.469201 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:41.469453 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:41.469542 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:41.472276 (+ 2734us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57475: BlockingRecv error: recv got EOF from 127.0.98.1:57475 (error 108)
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":75,"threads_started":1}
Sep 25 04:39:41 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:39:41 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:39:41.671643 9607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:41.665663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46371 (local address 127.0.98.2:48313)
0925 04:39:41.665988 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:41.666004 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:41.666110 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:41.666381 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:41.666384 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:41.666571 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:41.666666 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:41.666672 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:41.667645 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:41.667648 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:41.668236 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:41.668246 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:41.668734 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:41.668738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:41.668838 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:41.669417 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:41.669429 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:41.671488 (+ 2059us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46371: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":92,"threads_started":1}
W20260925 04:39:41.671828 9606 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:39:41.665937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48313 (local address 127.0.98.62:46371)
0925 04:39:41.666082 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:41.666087 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:41.666105 (+ 18us) server_negotiation.cc:407] Connection header received
0925 04:39:41.666204 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:41.666206 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:41.666249 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:41.666331 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:41.666768 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:41.667478 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:41.668370 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:41.668609 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:41.668707 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:41.671712 (+ 3005us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48313: BlockingRecv error: recv got EOF from 127.0.98.2:48313 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 25 04:39:41 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:39:41 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:39:41.876163 9608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:41.870590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46371 (local address 127.0.98.3:38281)
0925 04:39:41.870972 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:41.870990 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:41.871100 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:41.871338 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:41.871340 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:41.871545 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:41.871664 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:41.871669 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:41.872459 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:41.872461 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:41.873048 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:41.873056 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:41.873546 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:41.873549 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:41.873616 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:41.874117 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:41.874128 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:41.876037 (+ 1909us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46371: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":69,"threads_started":1}
W20260925 04:39:41.876348 9606 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:39:41.870647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38281 (local address 127.0.98.62:46371)
0925 04:39:41.870776 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:41.870780 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:41.871083 (+ 303us) server_negotiation.cc:407] Connection header received
0925 04:39:41.871184 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:41.871187 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:41.871239 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:41.871314 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:41.871781 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:41.872340 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:41.873179 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:41.873425 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:41.873516 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:41.876226 (+ 2710us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38281: BlockingRecv error: recv got EOF from 127.0.98.3:38281 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 25 04:39:42 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:39:42 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:39:42.479162 9609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:42.473155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46371 (local address 127.0.98.1:60827)
0925 04:39:42.473444 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:42.473458 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:42.473553 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:42.473912 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:42.473915 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:42.474128 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:42.474249 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:42.474256 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:42.475115 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:42.475117 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:42.475737 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:42.475746 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:42.476281 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:42.476285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:42.476367 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:42.476963 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:42.476976 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:42.479030 (+ 2054us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46371: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":82,"threads_started":1}
W20260925 04:39:42.479363 9610 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:39:42.473294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60827 (local address 127.0.98.62:46371)
0925 04:39:42.473669 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:42.473672 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:42.473686 (+ 14us) server_negotiation.cc:407] Connection header received
0925 04:39:42.473726 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:42.473729 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:42.473771 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:42.473860 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:42.474374 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:42.475005 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:42.475858 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:42.476112 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:42.476229 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:42.479225 (+ 2996us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:60827: BlockingRecv error: recv got EOF from 127.0.98.1:60827 (error 108)
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":71,"threads_started":1}
Sep 25 04:39:42 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:39:42 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:39:42.678387 9611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:42.672561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46371 (local address 127.0.98.2:58281)
0925 04:39:42.672894 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:42.672909 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:42.673031 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:42.673272 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:42.673275 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:42.673513 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:42.673609 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:42.673616 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:42.674372 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:42.674375 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:42.674952 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:42.674960 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:42.675402 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:42.675405 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:42.675485 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:42.676059 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:42.676071 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:42.678206 (+ 2135us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46371: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":108,"threads_started":1}
W20260925 04:39:42.678491 9610 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:39:42.672690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58281 (local address 127.0.98.62:46371)
0925 04:39:42.672888 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:42.672892 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:42.672986 (+ 94us) server_negotiation.cc:407] Connection header received
0925 04:39:42.673121 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:42.673124 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:42.673171 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:42.673237 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:42.673723 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:42.674257 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:42.675060 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:42.675277 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:42.675384 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:42.678383 (+ 2999us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58281: BlockingRecv error: recv got EOF from 127.0.98.2:58281 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 25 04:39:42 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:39:42 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:39:42.886806 9612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:42.877055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46371 (local address 127.0.98.3:55395)
0925 04:39:42.877388 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:42.877400 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:42.877512 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:42.877761 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:42.877764 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:42.877977 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:42.878112 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:42.878119 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:42.879090 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:42.879100 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:42.879988 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:42.879998 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:42.880448 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:42.880452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:42.880530 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:42.884616 (+ 4086us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:42.884630 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:42.886617 (+ 1987us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46371: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":111,"threads_started":1}
W20260925 04:39:42.886893 9610 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:39:42.877170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55395 (local address 127.0.98.62:46371)
0925 04:39:42.877316 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:42.877321 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:42.877468 (+ 147us) server_negotiation.cc:407] Connection header received
0925 04:39:42.877612 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:42.877615 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:42.877662 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:42.877725 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:42.878347 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:42.878947 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:42.880101 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:42.880341 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:42.880440 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:42.886788 (+ 6348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55395: BlockingRecv error: recv got EOF from 127.0.98.3:55395 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 25 04:39:43 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:39:43 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:39:43.486774 9613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:43.480101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46371 (local address 127.0.98.1:39707)
0925 04:39:43.480359 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:43.480373 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:43.480462 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:43.480676 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:43.480678 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:43.480846 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:43.480947 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:43.480952 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:43.481767 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:43.481771 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:43.482754 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:43.482767 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:43.483319 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:43.483323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:43.483415 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:43.483983 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:43.484026 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:43.486576 (+ 2550us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46371: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":79,"threads_started":1}
W20260925 04:39:43.486933 9614 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:39:43.480221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39707 (local address 127.0.98.62:46371)
0925 04:39:43.480467 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:43.480470 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:43.480485 (+ 15us) server_negotiation.cc:407] Connection header received
0925 04:39:43.480524 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:43.480526 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:43.480566 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:43.480654 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:43.481043 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:43.481613 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:43.482922 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:43.483175 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:43.483262 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:43.486777 (+ 3515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39707: BlockingRecv error: recv got EOF from 127.0.98.1:39707 (error 108)
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":81,"threads_started":1}
Sep 25 04:39:43 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:39:43 dist-test-slave-dc95 krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:39:43.685340 9615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:43.679454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46371 (local address 127.0.98.2:37641)
0925 04:39:43.679775 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:43.679789 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:43.679895 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:43.680247 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:43.680250 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:43.680457 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:43.680559 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:43.680565 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:43.681541 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:43.681546 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:43.682106 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:43.682116 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:43.682571 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:43.682575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:43.682653 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:43.683157 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:43.683181 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:43.685122 (+ 1941us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46371: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":84,"threads_started":1}
W20260925 04:39:43.685410 9614 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:39:43.679659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37641 (local address 127.0.98.62:46371)
0925 04:39:43.679810 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:43.679814 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:43.679847 (+ 33us) server_negotiation.cc:407] Connection header received
0925 04:39:43.679980 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:43.679983 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:43.680025 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:43.680182 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:43.680755 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:43.681427 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:43.682235 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:43.682464 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:43.682556 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:43.685306 (+ 2750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37641: BlockingRecv error: recv got EOF from 127.0.98.2:37641 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 04:39:43.762509 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 9139
I20260925 04:39:43.767524 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 9291
I20260925 04:39:43.772100 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 9422
I20260925 04:39:43.776312 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 9069
2026-09-25T04:39:43Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17951 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-25T04:39:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:39:43Z Disabled control of system clock
I20260925 04:39:43.804574 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311123478858-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311123478858-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311123478858-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44761
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44761
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 04:39:43.919569 9619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:39:43.919895 9619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:39:43.919930 9619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:39:43.924032 9619 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:39:43.924099 9619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:39:43.924113 9619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:39:43.924129 9619 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:39:43.924142 9619 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:39:43.929029 9619 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311123478858-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311123478858-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44761
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311123478858-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:39:43.930214 9619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:39:43.931738 9619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:39:43.940219 9627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:43.940156 9625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:43.940151 9624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:43.940584 9619 server_base.cc:1034] running on GCE node
I20260925 04:39:43.941177 9619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:39:43.942252 9619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:39:43.943409 9619 hybrid_clock.cc:648] HybridClock initialized: now 1790311183943384 us; error 42 us; skew 500 ppm
I20260925 04:39:43.945292 9619 webserver.cc:460] Webserver started at http://127.0.98.62:32945/ using document root <none> and password file <none>
I20260925 04:39:43.945855 9619 fs_manager.cc:359] Metadata directory not provided
I20260925 04:39:43.945915 9619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:39:43.946105 9619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:39:43.948190 9619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311123478858-392-0/minicluster-data/master-0/data/instance:
uuid: "a37cca786f4747998eec3225dd8949f9"
format_stamp: "Formatted at 2026-09-25 04:39:43 on dist-test-slave-dc95"
server_key: "09f42e68911c8dd3492eefed4011c478"
server_key_iv: "62ba04941b2b81cf7ea54c5cf09583b6"
server_key_version: "encryptionkey@0"
I20260925 04:39:43.948747 9619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311123478858-392-0/minicluster-data/master-0/wal/instance:
uuid: "a37cca786f4747998eec3225dd8949f9"
format_stamp: "Formatted at 2026-09-25 04:39:43 on dist-test-slave-dc95"
server_key: "09f42e68911c8dd3492eefed4011c478"
server_key_iv: "62ba04941b2b81cf7ea54c5cf09583b6"
server_key_version: "encryptionkey@0"
I20260925 04:39:43.952490 9619 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:39:43.955042 9633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:43.956422 9619 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:39:43.956562 9619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311123478858-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311123478858-392-0/minicluster-data/master-0/wal
uuid: "a37cca786f4747998eec3225dd8949f9"
format_stamp: "Formatted at 2026-09-25 04:39:43 on dist-test-slave-dc95"
server_key: "09f42e68911c8dd3492eefed4011c478"
server_key_iv: "62ba04941b2b81cf7ea54c5cf09583b6"
server_key_version: "encryptionkey@0"
I20260925 04:39:43.956703 9619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311123478858-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311123478858-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311123478858-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:39:43.969197 9619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:39:43.969942 9619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:39:43.970129 9619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:39:43.976989 9619 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44761
I20260925 04:39:43.976990 9685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44761 every 8 connection(s)
I20260925 04:39:43.977972 9619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
I20260925 04:39:43.980377 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 9619
I20260925 04:39:43.980485 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311123478858-392-0/minicluster-data/master-0/wal/instance
I20260925 04:39:43.980784 392 external_mini_cluster.cc:1428] Setting key 23de0442bb36a7f96304c5c76a3bee52
I20260925 04:39:43.981391 9686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:39:43.988297 9686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a37cca786f4747998eec3225dd8949f9: Bootstrap starting.
I20260925 04:39:43.990190 9689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:43.982238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42096 (local address 127.0.98.62:44761)
0925 04:39:43.982846 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:43.982862 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:43.982897 (+ 35us) server_negotiation.cc:407] Connection header received
0925 04:39:43.983568 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:43.983586 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:43.984044 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:43.984340 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:43.984700 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:43.985664 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:43.986814 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:43.987293 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:43.987406 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:43.988322 (+ 916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:43.988327 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:43.988340 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:39:43.988778 (+ 438us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:43.988836 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:39:43.988858 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:43.988939 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:43.989131 (+ 192us) server_negotiation.cc:299] Negotiation successful
0925 04:39:43.989418 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":117,"threads_started":1}
I20260925 04:39:43.990767 9686 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a37cca786f4747998eec3225dd8949f9: Neither blocks nor log segments found. Creating new log.
I20260925 04:39:43.991582 9686 log.cc:824] T 00000000000000000000000000000000 P a37cca786f4747998eec3225dd8949f9: Log is configured to *not* fsync() on all Append() calls
I20260925 04:39:43.993492 9686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a37cca786f4747998eec3225dd8949f9: No bootstrap required, opened a new log
I20260925 04:39:43.997133 9686 raft_consensus.cc:348] T 00000000000000000000000000000000 P a37cca786f4747998eec3225dd8949f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a37cca786f4747998eec3225dd8949f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44761 } }
I20260925 04:39:43.997401 9686 raft_consensus.cc:374] T 00000000000000000000000000000000 P a37cca786f4747998eec3225dd8949f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:39:43.997438 9686 raft_consensus.cc:729] T 00000000000000000000000000000000 P a37cca786f4747998eec3225dd8949f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a37cca786f4747998eec3225dd8949f9, State: Initialized, Role: FOLLOWER
I20260925 04:39:43.997915 9686 consensus_queue.cc:260] T 00000000000000000000000000000000 P a37cca786f4747998eec3225dd8949f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a37cca786f4747998eec3225dd8949f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44761 } }
I20260925 04:39:43.998044 9686 raft_consensus.cc:388] T 00000000000000000000000000000000 P a37cca786f4747998eec3225dd8949f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:39:43.998106 9686 raft_consensus.cc:482] T 00000000000000000000000000000000 P a37cca786f4747998eec3225dd8949f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:39:43.998205 9686 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a37cca786f4747998eec3225dd8949f9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:39:43.999260 9686 raft_consensus.cc:504] T 00000000000000000000000000000000 P a37cca786f4747998eec3225dd8949f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a37cca786f4747998eec3225dd8949f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44761 } }
I20260925 04:39:43.999632 9686 leader_election.cc:302] T 00000000000000000000000000000000 P a37cca786f4747998eec3225dd8949f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a37cca786f4747998eec3225dd8949f9; no voters:
I20260925 04:39:43.999959 9686 leader_election.cc:288] T 00000000000000000000000000000000 P a37cca786f4747998eec3225dd8949f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:39:44.000111 9691 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a37cca786f4747998eec3225dd8949f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:39:44.000334 9691 raft_consensus.cc:686] T 00000000000000000000000000000000 P a37cca786f4747998eec3225dd8949f9 [term 1 LEADER]: Becoming Leader. State: Replica: a37cca786f4747998eec3225dd8949f9, State: Running, Role: LEADER
I20260925 04:39:44.000638 9691 consensus_queue.cc:237] T 00000000000000000000000000000000 P a37cca786f4747998eec3225dd8949f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a37cca786f4747998eec3225dd8949f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44761 } }
I20260925 04:39:44.001183 9686 sys_catalog.cc:564] T 00000000000000000000000000000000 P a37cca786f4747998eec3225dd8949f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:39:44.002293 9693 sys_catalog.cc:455] T 00000000000000000000000000000000 P a37cca786f4747998eec3225dd8949f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a37cca786f4747998eec3225dd8949f9. Latest consensus state: current_term: 1 leader_uuid: "a37cca786f4747998eec3225dd8949f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a37cca786f4747998eec3225dd8949f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44761 } } }
I20260925 04:39:44.002314 9692 sys_catalog.cc:455] T 00000000000000000000000000000000 P a37cca786f4747998eec3225dd8949f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a37cca786f4747998eec3225dd8949f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a37cca786f4747998eec3225dd8949f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44761 } } }
I20260925 04:39:44.002415 9693 sys_catalog.cc:458] T 00000000000000000000000000000000 P a37cca786f4747998eec3225dd8949f9 [sys.catalog]: This master's current role is: LEADER
I20260925 04:39:44.002414 9692 sys_catalog.cc:458] T 00000000000000000000000000000000 P a37cca786f4747998eec3225dd8949f9 [sys.catalog]: This master's current role is: LEADER
I20260925 04:39:44.002843 9700 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:39:44.005848 9700 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:39:44.011897 9700 catalog_manager.cc:1347] Generated new cluster ID: 8754cacffe744e36b27b9637b1faf7ca
I20260925 04:39:44.012039 9700 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:39:44.032052 9700 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:39:44.033073 9700 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:39:44.051517 9700 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a37cca786f4747998eec3225dd8949f9: Generated new TSK 0
I20260925 04:39:44.052337 9700 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:39:44.057911 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311123478858-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311123478858-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311123478858-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44761
--builtin_ntp_servers=127.0.98.20:43813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 04:39:44.183274 9710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:39:44.183571 9710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:39:44.183637 9710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:39:44.187752 9710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:39:44.187824 9710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:39:44.187880 9710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 04:39:44.192741 9710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311123478858-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311123478858-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311123478858-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:39:44.193849 9710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:39:44.195304 9710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:39:44.205488 9718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:44.205489 9716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:44.205529 9715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:44.205746 9710 server_base.cc:1034] running on GCE node
I20260925 04:39:44.206300 9710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:39:44.206969 9710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:39:44.208143 9710 hybrid_clock.cc:648] HybridClock initialized: now 1790311184208114 us; error 48 us; skew 500 ppm
I20260925 04:39:44.210247 9710 webserver.cc:460] Webserver started at http://127.0.98.1:42185/ using document root <none> and password file <none>
I20260925 04:39:44.210856 9710 fs_manager.cc:359] Metadata directory not provided
I20260925 04:39:44.210928 9710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:39:44.211127 9710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:39:44.213311 9710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311123478858-392-0/minicluster-data/ts-0/data/instance:
uuid: "927188cf5a6647288e87ee1ee7ab89d7"
format_stamp: "Formatted at 2026-09-25 04:39:44 on dist-test-slave-dc95"
server_key: "6ffc02824e181167cc7b6b4b2c9535e4"
server_key_iv: "3121193cb41a394f0946b60ae1627255"
server_key_version: "encryptionkey@0"
I20260925 04:39:44.213873 9710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311123478858-392-0/minicluster-data/ts-0/wal/instance:
uuid: "927188cf5a6647288e87ee1ee7ab89d7"
format_stamp: "Formatted at 2026-09-25 04:39:44 on dist-test-slave-dc95"
server_key: "6ffc02824e181167cc7b6b4b2c9535e4"
server_key_iv: "3121193cb41a394f0946b60ae1627255"
server_key_version: "encryptionkey@0"
I20260925 04:39:44.217842 9710 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 04:39:44.220443 9724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:44.221788 9710 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:39:44.221908 9710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311123478858-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311123478858-392-0/minicluster-data/ts-0/wal
uuid: "927188cf5a6647288e87ee1ee7ab89d7"
format_stamp: "Formatted at 2026-09-25 04:39:44 on dist-test-slave-dc95"
server_key: "6ffc02824e181167cc7b6b4b2c9535e4"
server_key_iv: "3121193cb41a394f0946b60ae1627255"
server_key_version: "encryptionkey@0"
I20260925 04:39:44.222035 9710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311123478858-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311123478858-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311123478858-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:39:44.238632 9710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:39:44.239416 9710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:39:44.239593 9710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:39:44.240269 9710 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:39:44.241235 9710 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:39:44.241281 9710 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:44.241321 9710 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:39:44.241341 9710 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:44.249580 9710 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33483
I20260925 04:39:44.249604 9837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33483 every 8 connection(s)
I20260925 04:39:44.250480 9710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
I20260925 04:39:44.255023 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 9710
I20260925 04:39:44.255134 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311123478858-392-0/minicluster-data/ts-0/wal/instance
I20260925 04:39:44.255400 392 external_mini_cluster.cc:1428] Setting key 45d628a864323b4de651416106bf1fce
I20260925 04:39:44.257197 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311123478858-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311123478858-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311123478858-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44761
--builtin_ntp_servers=127.0.98.20:43813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 04:39:44.259706 9689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:44.252309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38109 (local address 127.0.98.62:44761)
0925 04:39:44.252443 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:44.252447 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:44.253474 (+ 1027us) server_negotiation.cc:407] Connection header received
0925 04:39:44.254818 (+ 1344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:44.254822 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:44.254880 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:44.254948 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:44.255972 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:44.256493 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:44.257672 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:44.257965 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:44.258066 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:44.259154 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:44.259157 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:44.259159 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:39:44.259171 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:44.259190 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:39:44.259197 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:44.259258 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:44.259460 (+ 202us) server_negotiation.cc:299] Negotiation successful
0925 04:39:44.259533 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 04:39:44.260547 9840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:44.252690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44761 (local address 127.0.98.1:38109)
0925 04:39:44.253329 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:44.253370 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:44.254583 (+ 1213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:44.255141 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:44.255150 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:44.255261 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:44.255833 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:44.255848 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:44.256628 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:44.256630 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:44.257560 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:44.257569 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:44.258096 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:44.258098 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:44.258241 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:44.258842 (+ 601us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:39:44.258844 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:44.258902 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:39:44.259000 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:39:44.259279 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:39:44.259289 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:39:44.259294 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 04:39:44.259425 (+ 131us) client_negotiation.cc:240] Negotiation successful
0925 04:39:44.259728 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":105,"threads_started":1}
I20260925 04:39:44.261723 9838 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44761
I20260925 04:39:44.261955 9838 heartbeater.cc:464] Registering TS with master...
I20260925 04:39:44.262470 9838 heartbeater.cc:511] Master 127.0.98.62:44761 requested a full tablet report, sending...
I20260925 04:39:44.263954 9648 ts_manager.cc:194] Registered new tserver with Master: 927188cf5a6647288e87ee1ee7ab89d7 (127.0.98.1:33483)
I20260925 04:39:44.265080 9648 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:38109
W20260925 04:39:44.370774 9841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:39:44.371066 9841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:39:44.371119 9841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:39:44.375348 9841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:39:44.375428 9841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:39:44.375505 9841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 04:39:44.380602 9841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311123478858-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311123478858-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311123478858-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:39:44.381731 9841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:39:44.383142 9841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:39:44.393185 9849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:44.393162 9846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:44.393587 9847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:44.393566 9841 server_base.cc:1034] running on GCE node
I20260925 04:39:44.394083 9841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:39:44.394778 9841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:39:44.395936 9841 hybrid_clock.cc:648] HybridClock initialized: now 1790311184395915 us; error 34 us; skew 500 ppm
I20260925 04:39:44.398041 9841 webserver.cc:460] Webserver started at http://127.0.98.2:39251/ using document root <none> and password file <none>
I20260925 04:39:44.398654 9841 fs_manager.cc:359] Metadata directory not provided
I20260925 04:39:44.398721 9841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:39:44.398916 9841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:39:44.401202 9841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311123478858-392-0/minicluster-data/ts-1/data/instance:
uuid: "3149bbbad0524203b79dc15a34ef00a3"
format_stamp: "Formatted at 2026-09-25 04:39:44 on dist-test-slave-dc95"
server_key: "6ff0cf2bbdcabb6f76e593bd97440643"
server_key_iv: "af24eed9382c8a3e1b09c05298362751"
server_key_version: "encryptionkey@0"
I20260925 04:39:44.401799 9841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311123478858-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3149bbbad0524203b79dc15a34ef00a3"
format_stamp: "Formatted at 2026-09-25 04:39:44 on dist-test-slave-dc95"
server_key: "6ff0cf2bbdcabb6f76e593bd97440643"
server_key_iv: "af24eed9382c8a3e1b09c05298362751"
server_key_version: "encryptionkey@0"
I20260925 04:39:44.405701 9841 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 04:39:44.408423 9855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:44.409828 9841 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:39:44.409961 9841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311123478858-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311123478858-392-0/minicluster-data/ts-1/wal
uuid: "3149bbbad0524203b79dc15a34ef00a3"
format_stamp: "Formatted at 2026-09-25 04:39:44 on dist-test-slave-dc95"
server_key: "6ff0cf2bbdcabb6f76e593bd97440643"
server_key_iv: "af24eed9382c8a3e1b09c05298362751"
server_key_version: "encryptionkey@0"
I20260925 04:39:44.410081 9841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311123478858-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311123478858-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311123478858-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:39:44.425817 9841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:39:44.426594 9841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:39:44.426770 9841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:39:44.427366 9841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:39:44.428356 9841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:39:44.428406 9841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:44.428458 9841 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:39:44.428486 9841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:44.437428 9841 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45167
I20260925 04:39:44.437449 9968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45167 every 8 connection(s)
I20260925 04:39:44.438388 9841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
I20260925 04:39:44.444509 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 9841
I20260925 04:39:44.444617 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311123478858-392-0/minicluster-data/ts-1/wal/instance
I20260925 04:39:44.444934 392 external_mini_cluster.cc:1428] Setting key 45dae50197e091455ccfb997bd6e2c69
I20260925 04:39:44.446868 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311123478858-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311123478858-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311123478858-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44761
--builtin_ntp_servers=127.0.98.20:43813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 04:39:44.447793 9689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:44.440264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48153 (local address 127.0.98.62:44761)
0925 04:39:44.440426 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:44.440431 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:44.441479 (+ 1048us) server_negotiation.cc:407] Connection header received
0925 04:39:44.442839 (+ 1360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:44.442842 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:44.442893 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:44.442969 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:44.443989 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:44.444522 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:44.445818 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:44.446116 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:44.446227 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:44.447321 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:44.447324 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:44.447327 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:39:44.447339 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:44.447359 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:39:44.447366 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:44.447423 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:44.447568 (+ 145us) server_negotiation.cc:299] Negotiation successful
0925 04:39:44.447653 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 04:39:44.448758 9971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:44.440676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44761 (local address 127.0.98.2:48153)
0925 04:39:44.441346 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:44.441388 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:44.442648 (+ 1260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:44.443130 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:44.443137 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:44.443225 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:44.443832 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:44.443855 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:44.444678 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:44.444681 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:44.445702 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:44.445712 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:44.446279 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:44.446284 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:44.446443 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:44.447058 (+ 615us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:39:44.447060 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:44.447105 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:39:44.447173 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:39:44.447433 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:39:44.447442 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:39:44.447448 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 04:39:44.447559 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 04:39:44.447898 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":122,"threads_started":1}
I20260925 04:39:44.449996 9969 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44761
I20260925 04:39:44.450259 9969 heartbeater.cc:464] Registering TS with master...
I20260925 04:39:44.450802 9969 heartbeater.cc:511] Master 127.0.98.62:44761 requested a full tablet report, sending...
I20260925 04:39:44.452046 9648 ts_manager.cc:194] Registered new tserver with Master: 3149bbbad0524203b79dc15a34ef00a3 (127.0.98.2:45167)
I20260925 04:39:44.452886 9648 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:48153
W20260925 04:39:44.562026 9972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:39:44.562361 9972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:39:44.562405 9972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:39:44.566594 9972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:39:44.566668 9972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:39:44.566730 9972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 04:39:44.571744 9972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311123478858-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311123478858-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311123478858-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:39:44.572891 9972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:39:44.574282 9972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:39:44.583913 9980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:44.583952 9972 server_base.cc:1034] running on GCE node
W20260925 04:39:44.583921 9978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:44.583901 9977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:44.584554 9972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:39:44.585220 9972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:39:44.586383 9972 hybrid_clock.cc:648] HybridClock initialized: now 1790311184586354 us; error 43 us; skew 500 ppm
I20260925 04:39:44.588254 9972 webserver.cc:460] Webserver started at http://127.0.98.3:33103/ using document root <none> and password file <none>
I20260925 04:39:44.588819 9972 fs_manager.cc:359] Metadata directory not provided
I20260925 04:39:44.588887 9972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:39:44.589093 9972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:39:44.591145 9972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311123478858-392-0/minicluster-data/ts-2/data/instance:
uuid: "4229b7631cc841209f9455f76ea8db5c"
format_stamp: "Formatted at 2026-09-25 04:39:44 on dist-test-slave-dc95"
server_key: "9001e800017a94f35671d410de341092"
server_key_iv: "982ea3052db861706e017e6e0ae0468c"
server_key_version: "encryptionkey@0"
I20260925 04:39:44.591777 9972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311123478858-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4229b7631cc841209f9455f76ea8db5c"
format_stamp: "Formatted at 2026-09-25 04:39:44 on dist-test-slave-dc95"
server_key: "9001e800017a94f35671d410de341092"
server_key_iv: "982ea3052db861706e017e6e0ae0468c"
server_key_version: "encryptionkey@0"
I20260925 04:39:44.595775 9972 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:39:44.598809 9986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:44.600204 9972 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:39:44.600317 9972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311123478858-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311123478858-392-0/minicluster-data/ts-2/wal
uuid: "4229b7631cc841209f9455f76ea8db5c"
format_stamp: "Formatted at 2026-09-25 04:39:44 on dist-test-slave-dc95"
server_key: "9001e800017a94f35671d410de341092"
server_key_iv: "982ea3052db861706e017e6e0ae0468c"
server_key_version: "encryptionkey@0"
I20260925 04:39:44.600425 9972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311123478858-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311123478858-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311123478858-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:39:44.618577 9972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:39:44.619258 9972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:39:44.619417 9972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:39:44.620023 9972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:39:44.620944 9972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:39:44.620991 9972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:44.621032 9972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:39:44.621052 9972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:44.629597 9972 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41907
I20260925 04:39:44.629622 10099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41907 every 8 connection(s)
I20260925 04:39:44.630558 9972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
I20260925 04:39:44.633796 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 9972
I20260925 04:39:44.633920 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311123478858-392-0/minicluster-data/ts-2/wal/instance
I20260925 04:39:44.634219 392 external_mini_cluster.cc:1428] Setting key ba2bc22a2b50bed97c5bfe3af41e3ab8
I20260925 04:39:44.640163 9689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:44.632379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45863 (local address 127.0.98.62:44761)
0925 04:39:44.632539 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:44.632544 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:44.633708 (+ 1164us) server_negotiation.cc:407] Connection header received
0925 04:39:44.635322 (+ 1614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:44.635325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:44.635385 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:44.635462 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:44.636545 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:44.637086 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:44.638282 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:44.638515 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:44.638605 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:44.639745 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:44.639748 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:44.639750 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:39:44.639766 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:44.639787 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:39:44.639794 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:44.639852 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:44.639980 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 04:39:44.640041 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 04:39:44.641110 10102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:44.632824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44761 (local address 127.0.98.3:45863)
0925 04:39:44.633515 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:44.633573 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:44.635080 (+ 1507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:44.635685 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:44.635694 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:44.635791 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:44.636412 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:44.636429 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:44.637181 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:44.637183 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:44.638119 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:44.638129 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:44.638650 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:44.638653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:44.638820 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:44.639416 (+ 596us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:39:44.639418 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:44.639466 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:39:44.639543 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:39:44.639862 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:39:44.639870 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:39:44.639874 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 04:39:44.639976 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 04:39:44.640246 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":111,"threads_started":1}
I20260925 04:39:44.642320 10100 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44761
I20260925 04:39:44.642575 10100 heartbeater.cc:464] Registering TS with master...
I20260925 04:39:44.643091 10100 heartbeater.cc:511] Master 127.0.98.62:44761 requested a full tablet report, sending...
I20260925 04:39:44.644121 9648 ts_manager.cc:194] Registered new tserver with Master: 4229b7631cc841209f9455f76ea8db5c (127.0.98.3:41907)
I20260925 04:39:44.644654 9648 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:45863
I20260925 04:39:44.647953 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 04:39:44.650478 9689 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:39:44.649707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42108 (local address 127.0.98.62:44761)
0925 04:39:44.649851 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:44.649856 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:44.649964 (+ 108us) server_negotiation.cc:407] Connection header received
0925 04:39:44.650058 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:44.650060 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:44.650099 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:44.650159 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:44.650348 (+ 189us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42108: BlockingRecv error: recv got EOF from 127.0.0.1:42108 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 04:39:44.650627 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 9710
I20260925 04:39:44.655498 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 9841
I20260925 04:39:44.660007 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 9972
I20260925 04:39:44.664506 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 9619
2026-09-25T04:39:44Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (887 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-25T04:39:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:39:44Z Disabled control of system clock
I20260925 04:39:44.691756 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311123478858-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311123478858-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311123478858-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34787
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34787
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 04:39:44.802832 10111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:39:44.803189 10111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:39:44.803237 10111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:39:44.807930 10111 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:39:44.808013 10111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:39:44.808028 10111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:39:44.808045 10111 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:39:44.808060 10111 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:39:44.813338 10111 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311123478858-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311123478858-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34787
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311123478858-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:39:44.814472 10111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:39:44.815964 10111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:39:44.824584 10116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:44.824712 10119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:44.824712 10117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:44.824776 10111 server_base.cc:1034] running on GCE node
I20260925 04:39:44.825508 10111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:39:44.826668 10111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:39:44.827831 10111 hybrid_clock.cc:648] HybridClock initialized: now 1790311184827806 us; error 38 us; skew 500 ppm
I20260925 04:39:44.829747 10111 webserver.cc:460] Webserver started at http://127.0.98.62:45299/ using document root <none> and password file <none>
I20260925 04:39:44.830328 10111 fs_manager.cc:359] Metadata directory not provided
I20260925 04:39:44.830396 10111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:39:44.830590 10111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:39:44.832921 10111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311123478858-392-0/minicluster-data/master-0/data/instance:
uuid: "fdc88c9709f945fca808a1e4785da7df"
format_stamp: "Formatted at 2026-09-25 04:39:44 on dist-test-slave-dc95"
server_key: "2e5fc278d2d287e253658e59b947a63c"
server_key_iv: "5098ab66f6b5da48261ac6462264c1c2"
server_key_version: "encryptionkey@0"
I20260925 04:39:44.833520 10111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311123478858-392-0/minicluster-data/master-0/wal/instance:
uuid: "fdc88c9709f945fca808a1e4785da7df"
format_stamp: "Formatted at 2026-09-25 04:39:44 on dist-test-slave-dc95"
server_key: "2e5fc278d2d287e253658e59b947a63c"
server_key_iv: "5098ab66f6b5da48261ac6462264c1c2"
server_key_version: "encryptionkey@0"
I20260925 04:39:44.837565 10111 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 04:39:44.840281 10125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:44.841683 10111 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:39:44.841820 10111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311123478858-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311123478858-392-0/minicluster-data/master-0/wal
uuid: "fdc88c9709f945fca808a1e4785da7df"
format_stamp: "Formatted at 2026-09-25 04:39:44 on dist-test-slave-dc95"
server_key: "2e5fc278d2d287e253658e59b947a63c"
server_key_iv: "5098ab66f6b5da48261ac6462264c1c2"
server_key_version: "encryptionkey@0"
I20260925 04:39:44.841940 10111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311123478858-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311123478858-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311123478858-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:39:44.846675 10111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:39:44.847326 10111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:39:44.847482 10111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:39:44.854133 10111 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34787
I20260925 04:39:44.854143 10177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34787 every 8 connection(s)
I20260925 04:39:44.855077 10111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
I20260925 04:39:44.857558 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 10111
I20260925 04:39:44.857673 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311123478858-392-0/minicluster-data/master-0/wal/instance
I20260925 04:39:44.857933 392 external_mini_cluster.cc:1428] Setting key 0475e852f8f8adc8794fa473936d8c16
I20260925 04:39:44.858386 10178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:39:44.864306 10181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:44.859665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55126 (local address 127.0.98.62:34787)
0925 04:39:44.860349 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:44.860369 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:44.860414 (+ 45us) server_negotiation.cc:407] Connection header received
0925 04:39:44.861133 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:44.861159 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:44.861627 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:44.862031 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:44.862748 (+ 717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:44.862753 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:44.862779 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:39:44.862790 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:44.862852 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:39:44.862873 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:44.863018 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:44.863091 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 04:39:44.863348 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":114,"threads_started":1}
I20260925 04:39:44.865856 10178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdc88c9709f945fca808a1e4785da7df: Bootstrap starting.
I20260925 04:39:44.868132 10178 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fdc88c9709f945fca808a1e4785da7df: Neither blocks nor log segments found. Creating new log.
I20260925 04:39:44.868851 10178 log.cc:824] T 00000000000000000000000000000000 P fdc88c9709f945fca808a1e4785da7df: Log is configured to *not* fsync() on all Append() calls
I20260925 04:39:44.870721 10178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdc88c9709f945fca808a1e4785da7df: No bootstrap required, opened a new log
I20260925 04:39:44.874396 10178 raft_consensus.cc:348] T 00000000000000000000000000000000 P fdc88c9709f945fca808a1e4785da7df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc88c9709f945fca808a1e4785da7df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34787 } }
I20260925 04:39:44.874655 10178 raft_consensus.cc:374] T 00000000000000000000000000000000 P fdc88c9709f945fca808a1e4785da7df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:39:44.874714 10178 raft_consensus.cc:729] T 00000000000000000000000000000000 P fdc88c9709f945fca808a1e4785da7df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc88c9709f945fca808a1e4785da7df, State: Initialized, Role: FOLLOWER
I20260925 04:39:44.875138 10178 consensus_queue.cc:260] T 00000000000000000000000000000000 P fdc88c9709f945fca808a1e4785da7df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc88c9709f945fca808a1e4785da7df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34787 } }
I20260925 04:39:44.875276 10178 raft_consensus.cc:388] T 00000000000000000000000000000000 P fdc88c9709f945fca808a1e4785da7df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:39:44.875339 10178 raft_consensus.cc:482] T 00000000000000000000000000000000 P fdc88c9709f945fca808a1e4785da7df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:39:44.875442 10178 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fdc88c9709f945fca808a1e4785da7df [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:39:44.876488 10178 raft_consensus.cc:504] T 00000000000000000000000000000000 P fdc88c9709f945fca808a1e4785da7df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc88c9709f945fca808a1e4785da7df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34787 } }
I20260925 04:39:44.876883 10178 leader_election.cc:302] T 00000000000000000000000000000000 P fdc88c9709f945fca808a1e4785da7df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fdc88c9709f945fca808a1e4785da7df; no voters:
I20260925 04:39:44.877200 10178 leader_election.cc:288] T 00000000000000000000000000000000 P fdc88c9709f945fca808a1e4785da7df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:39:44.877315 10183 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fdc88c9709f945fca808a1e4785da7df [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:39:44.877558 10183 raft_consensus.cc:686] T 00000000000000000000000000000000 P fdc88c9709f945fca808a1e4785da7df [term 1 LEADER]: Becoming Leader. State: Replica: fdc88c9709f945fca808a1e4785da7df, State: Running, Role: LEADER
I20260925 04:39:44.877926 10183 consensus_queue.cc:237] T 00000000000000000000000000000000 P fdc88c9709f945fca808a1e4785da7df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc88c9709f945fca808a1e4785da7df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34787 } }
I20260925 04:39:44.878559 10178 sys_catalog.cc:564] T 00000000000000000000000000000000 P fdc88c9709f945fca808a1e4785da7df [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:39:44.879702 10184 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdc88c9709f945fca808a1e4785da7df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fdc88c9709f945fca808a1e4785da7df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc88c9709f945fca808a1e4785da7df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34787 } } }
I20260925 04:39:44.879846 10184 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdc88c9709f945fca808a1e4785da7df [sys.catalog]: This master's current role is: LEADER
I20260925 04:39:44.880138 10185 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdc88c9709f945fca808a1e4785da7df [sys.catalog]: SysCatalogTable state changed. Reason: New leader fdc88c9709f945fca808a1e4785da7df. Latest consensus state: current_term: 1 leader_uuid: "fdc88c9709f945fca808a1e4785da7df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc88c9709f945fca808a1e4785da7df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34787 } } }
I20260925 04:39:44.880229 10185 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdc88c9709f945fca808a1e4785da7df [sys.catalog]: This master's current role is: LEADER
I20260925 04:39:44.880263 10192 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:39:44.883248 10192 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:39:44.888805 10192 catalog_manager.cc:1347] Generated new cluster ID: 2d7045cbbf7e48ba99c56fd35fe2f83b
I20260925 04:39:44.888888 10192 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:39:44.895839 10192 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:39:44.896158 10192 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:39:44.907739 10192 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fdc88c9709f945fca808a1e4785da7df: Generated new TSK 0
I20260925 04:39:44.908533 10192 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:39:44.918239 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311123478858-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311123478858-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311123478858-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34787
--builtin_ntp_servers=127.0.98.20:43267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 04:39:45.029047 10202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:39:45.029336 10202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:39:45.029383 10202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:39:45.033646 10202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:39:45.033723 10202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:39:45.033798 10202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 04:39:45.038916 10202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311123478858-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311123478858-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311123478858-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:39:45.040108 10202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:39:45.041605 10202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:39:45.051723 10210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:45.051682 10207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:45.051685 10208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:45.051932 10202 server_base.cc:1034] running on GCE node
I20260925 04:39:45.052363 10202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:39:45.053004 10202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:39:45.054157 10202 hybrid_clock.cc:648] HybridClock initialized: now 1790311185054138 us; error 35 us; skew 500 ppm
I20260925 04:39:45.056175 10202 webserver.cc:460] Webserver started at http://127.0.98.1:44365/ using document root <none> and password file <none>
I20260925 04:39:45.056772 10202 fs_manager.cc:359] Metadata directory not provided
I20260925 04:39:45.056842 10202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:39:45.057035 10202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:39:45.059165 10202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311123478858-392-0/minicluster-data/ts-0/data/instance:
uuid: "ebc940e5bfe145eabafef22f94323999"
format_stamp: "Formatted at 2026-09-25 04:39:45 on dist-test-slave-dc95"
server_key: "b44a19283ce76158d946b31f066296e7"
server_key_iv: "eb569951b34871641a0e6bed1f9e2c64"
server_key_version: "encryptionkey@0"
I20260925 04:39:45.059773 10202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311123478858-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ebc940e5bfe145eabafef22f94323999"
format_stamp: "Formatted at 2026-09-25 04:39:45 on dist-test-slave-dc95"
server_key: "b44a19283ce76158d946b31f066296e7"
server_key_iv: "eb569951b34871641a0e6bed1f9e2c64"
server_key_version: "encryptionkey@0"
I20260925 04:39:45.063689 10202 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:39:45.066380 10216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:45.067778 10202 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:39:45.067906 10202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311123478858-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311123478858-392-0/minicluster-data/ts-0/wal
uuid: "ebc940e5bfe145eabafef22f94323999"
format_stamp: "Formatted at 2026-09-25 04:39:45 on dist-test-slave-dc95"
server_key: "b44a19283ce76158d946b31f066296e7"
server_key_iv: "eb569951b34871641a0e6bed1f9e2c64"
server_key_version: "encryptionkey@0"
I20260925 04:39:45.068020 10202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311123478858-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311123478858-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311123478858-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:39:45.072808 10202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:39:45.073453 10202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:39:45.073618 10202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:39:45.074162 10202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:39:45.075110 10202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:39:45.075158 10202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:45.075218 10202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:39:45.075246 10202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:45.084229 10202 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43591
I20260925 04:39:45.084257 10329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43591 every 8 connection(s)
I20260925 04:39:45.085166 10202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
I20260925 04:39:45.091228 10181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:45.086990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42215 (local address 127.0.98.62:34787)
0925 04:39:45.087155 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:45.087160 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:45.088267 (+ 1107us) server_negotiation.cc:407] Connection header received
0925 04:39:45.089545 (+ 1278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:45.089547 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:45.089584 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:45.089645 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:45.090803 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:45.090806 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:45.090809 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:39:45.090828 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:45.090848 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:39:45.090856 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:45.090936 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:45.091085 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 04:39:45.091120 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 04:39:45.092186 10332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:45.087413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34787 (local address 127.0.98.1:42215)
0925 04:39:45.088116 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:45.088162 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:45.089345 (+ 1183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:45.089845 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:45.089855 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:45.089954 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:45.090518 (+ 564us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:39:45.090519 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:45.090571 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:39:45.090649 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:39:45.090956 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:39:45.090964 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:39:45.090970 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 04:39:45.091128 (+ 158us) client_negotiation.cc:240] Negotiation successful
0925 04:39:45.091356 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":148,"threads_started":1}
I20260925 04:39:45.093382 10330 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34787
I20260925 04:39:45.093595 10330 heartbeater.cc:464] Registering TS with master...
I20260925 04:39:45.093971 10330 heartbeater.cc:511] Master 127.0.98.62:34787 requested a full tablet report, sending...
I20260925 04:39:45.094661 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 10202
I20260925 04:39:45.094769 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311123478858-392-0/minicluster-data/ts-0/wal/instance
I20260925 04:39:45.095086 392 external_mini_cluster.cc:1428] Setting key 9e60330216cd4b72f36c99352c48bccd
I20260925 04:39:45.095512 10142 ts_manager.cc:194] Registered new tserver with Master: ebc940e5bfe145eabafef22f94323999 (127.0.98.1:43591)
I20260925 04:39:45.097235 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311123478858-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311123478858-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311123478858-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34787
--builtin_ntp_servers=127.0.98.20:43267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 04:39:45.209426 10333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:39:45.209721 10333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:39:45.209764 10333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:39:45.213959 10333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:39:45.214026 10333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:39:45.214087 10333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 04:39:45.219061 10333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311123478858-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311123478858-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311123478858-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:39:45.220175 10333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:39:45.221563 10333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:39:45.231197 10341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:45.231245 10333 server_base.cc:1034] running on GCE node
W20260925 04:39:45.231177 10338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:45.231178 10339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:45.231861 10333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:39:45.232543 10333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:39:45.233712 10333 hybrid_clock.cc:648] HybridClock initialized: now 1790311185233679 us; error 51 us; skew 500 ppm
I20260925 04:39:45.235705 10333 webserver.cc:460] Webserver started at http://127.0.98.2:46845/ using document root <none> and password file <none>
I20260925 04:39:45.236306 10333 fs_manager.cc:359] Metadata directory not provided
I20260925 04:39:45.236373 10333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:39:45.236568 10333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:39:45.238725 10333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311123478858-392-0/minicluster-data/ts-1/data/instance:
uuid: "4e3f1abdb09e4879bf537524dee76112"
format_stamp: "Formatted at 2026-09-25 04:39:45 on dist-test-slave-dc95"
server_key: "a1a644cc4ff4e05ad0c3950bc06a8462"
server_key_iv: "533600cd70337869a51b28654bd06b52"
server_key_version: "encryptionkey@0"
I20260925 04:39:45.239328 10333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311123478858-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4e3f1abdb09e4879bf537524dee76112"
format_stamp: "Formatted at 2026-09-25 04:39:45 on dist-test-slave-dc95"
server_key: "a1a644cc4ff4e05ad0c3950bc06a8462"
server_key_iv: "533600cd70337869a51b28654bd06b52"
server_key_version: "encryptionkey@0"
I20260925 04:39:45.243276 10333 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260925 04:39:45.245873 10347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:45.247138 10333 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:39:45.247242 10333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311123478858-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311123478858-392-0/minicluster-data/ts-1/wal
uuid: "4e3f1abdb09e4879bf537524dee76112"
format_stamp: "Formatted at 2026-09-25 04:39:45 on dist-test-slave-dc95"
server_key: "a1a644cc4ff4e05ad0c3950bc06a8462"
server_key_iv: "533600cd70337869a51b28654bd06b52"
server_key_version: "encryptionkey@0"
I20260925 04:39:45.247357 10333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311123478858-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311123478858-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311123478858-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:39:45.252357 10333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:39:45.253036 10333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:39:45.253211 10333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:39:45.253789 10333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:39:45.254738 10333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:39:45.254786 10333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:45.254838 10333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:39:45.254863 10333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:45.263255 10333 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32921
I20260925 04:39:45.263293 10460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32921 every 8 connection(s)
I20260925 04:39:45.264235 10333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
I20260925 04:39:45.270665 10181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:45.266122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38089 (local address 127.0.98.62:34787)
0925 04:39:45.266288 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:45.266293 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:45.267460 (+ 1167us) server_negotiation.cc:407] Connection header received
0925 04:39:45.268850 (+ 1390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:45.268853 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:45.268898 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:45.268979 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:45.270213 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:45.270216 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:45.270218 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:39:45.270231 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:45.270252 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:39:45.270259 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:45.270355 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:45.270505 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 04:39:45.270545 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 04:39:45.271813 10463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:45.266572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34787 (local address 127.0.98.2:38089)
0925 04:39:45.267300 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:45.267344 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:45.268594 (+ 1250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:45.269154 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:45.269161 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:45.269292 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:45.269936 (+ 644us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:39:45.269938 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:45.269982 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:39:45.270058 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:39:45.270373 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:39:45.270382 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:39:45.270387 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 04:39:45.270558 (+ 171us) client_negotiation.cc:240] Negotiation successful
0925 04:39:45.270796 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":122,"threads_started":1}
I20260925 04:39:45.273059 10461 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34787
I20260925 04:39:45.273295 10461 heartbeater.cc:464] Registering TS with master...
I20260925 04:39:45.273612 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 10333
I20260925 04:39:45.273717 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311123478858-392-0/minicluster-data/ts-1/wal/instance
I20260925 04:39:45.273741 10461 heartbeater.cc:511] Master 127.0.98.62:34787 requested a full tablet report, sending...
I20260925 04:39:45.274044 392 external_mini_cluster.cc:1428] Setting key 8b8c6ee665deca70fae9bf21ea40ae48
I20260925 04:39:45.274957 10142 ts_manager.cc:194] Registered new tserver with Master: 4e3f1abdb09e4879bf537524dee76112 (127.0.98.2:32921)
I20260925 04:39:45.276144 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311123478858-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311123478858-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311123478858-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34787
--builtin_ntp_servers=127.0.98.20:43267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 04:39:45.388164 10464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:39:45.388485 10464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:39:45.388545 10464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:39:45.392741 10464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:39:45.392809 10464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:39:45.392889 10464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 04:39:45.397979 10464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311123478858-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311123478858-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311123478858-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:39:45.399132 10464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:39:45.400959 10464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:39:45.412415 10470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:45.412390 10472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:45.412446 10469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:45.412603 10464 server_base.cc:1034] running on GCE node
I20260925 04:39:45.413230 10464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:39:45.413884 10464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:39:45.415045 10464 hybrid_clock.cc:648] HybridClock initialized: now 1790311185415025 us; error 38 us; skew 500 ppm
I20260925 04:39:45.416908 10464 webserver.cc:460] Webserver started at http://127.0.98.3:37217/ using document root <none> and password file <none>
I20260925 04:39:45.417451 10464 fs_manager.cc:359] Metadata directory not provided
I20260925 04:39:45.417527 10464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:39:45.417721 10464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:39:45.419842 10464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311123478858-392-0/minicluster-data/ts-2/data/instance:
uuid: "f89c144021ac4d7bae6686eeb483901d"
format_stamp: "Formatted at 2026-09-25 04:39:45 on dist-test-slave-dc95"
server_key: "127d32ff24c619e6434a88ca3655d14e"
server_key_iv: "15a771f733b326d4482ee39f4aecc463"
server_key_version: "encryptionkey@0"
I20260925 04:39:45.420390 10464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311123478858-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f89c144021ac4d7bae6686eeb483901d"
format_stamp: "Formatted at 2026-09-25 04:39:45 on dist-test-slave-dc95"
server_key: "127d32ff24c619e6434a88ca3655d14e"
server_key_iv: "15a771f733b326d4482ee39f4aecc463"
server_key_version: "encryptionkey@0"
I20260925 04:39:45.424233 10464 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:39:45.426754 10478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:45.428143 10464 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:39:45.428253 10464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311123478858-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311123478858-392-0/minicluster-data/ts-2/wal
uuid: "f89c144021ac4d7bae6686eeb483901d"
format_stamp: "Formatted at 2026-09-25 04:39:45 on dist-test-slave-dc95"
server_key: "127d32ff24c619e6434a88ca3655d14e"
server_key_iv: "15a771f733b326d4482ee39f4aecc463"
server_key_version: "encryptionkey@0"
I20260925 04:39:45.428359 10464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311123478858-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311123478858-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311123478858-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:39:45.433025 10464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:39:45.433636 10464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:39:45.433789 10464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:39:45.434331 10464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:39:45.435220 10464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:39:45.435266 10464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:45.435317 10464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:39:45.435343 10464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:45.443625 10464 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42841
I20260925 04:39:45.443681 10591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42841 every 8 connection(s)
I20260925 04:39:45.444558 10464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
I20260925 04:39:45.450604 10181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:45.446435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41967 (local address 127.0.98.62:34787)
0925 04:39:45.446585 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:45.446589 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:45.447676 (+ 1087us) server_negotiation.cc:407] Connection header received
0925 04:39:45.449070 (+ 1394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:45.449073 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:45.449113 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:45.449225 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:45.450251 (+ 1026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:45.450254 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:45.450257 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:39:45.450269 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:45.450290 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:39:45.450297 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:45.450386 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:45.450452 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 04:39:45.450491 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 04:39:45.451563 10594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:45.446841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34787 (local address 127.0.98.3:41967)
0925 04:39:45.447510 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:45.447550 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:45.448812 (+ 1262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:45.449385 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:45.449392 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:45.449486 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:45.450002 (+ 516us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:39:45.450003 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:45.450045 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:39:45.450118 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:39:45.450355 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:39:45.450359 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:39:45.450362 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 04:39:45.450488 (+ 126us) client_negotiation.cc:240] Negotiation successful
0925 04:39:45.450726 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":115,"threads_started":1}
I20260925 04:39:45.452425 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 10464
I20260925 04:39:45.452533 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311123478858-392-0/minicluster-data/ts-2/wal/instance
I20260925 04:39:45.452811 392 external_mini_cluster.cc:1428] Setting key 385718d50eec33cc6960a2e01c7ffb64
I20260925 04:39:45.452915 10592 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34787
I20260925 04:39:45.453164 10592 heartbeater.cc:464] Registering TS with master...
I20260925 04:39:45.453670 10592 heartbeater.cc:511] Master 127.0.98.62:34787 requested a full tablet report, sending...
I20260925 04:39:45.454700 10141 ts_manager.cc:194] Registered new tserver with Master: f89c144021ac4d7bae6686eeb483901d (127.0.98.3:42841)
I20260925 04:39:45.466902 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 04:39:45.469799 10181 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:39:45.468834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55138 (local address 127.0.98.62:34787)
0925 04:39:45.468981 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:45.468985 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:45.469096 (+ 111us) server_negotiation.cc:407] Connection header received
0925 04:39:45.469199 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:45.469201 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:45.469235 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:45.469299 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:45.469633 (+ 334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55138: BlockingRecv error: recv got EOF from 127.0.0.1:55138 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 04:39:45.469929 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 10202
I20260925 04:39:45.474757 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 10333
I20260925 04:39:45.479463 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 10464
I20260925 04:39:45.484187 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 10111
2026-09-25T04:39:45Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (819 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:39:45 dist-test-slave-dc95 krb5kdc[10602](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:39:45 dist-test-slave-dc95 krb5kdc[10602](info): set up 2 sockets
Sep 25 04:39:45 dist-test-slave-dc95 krb5kdc[10602](info): commencing operation
krb5kdc: starting...
W20260925 04:39:46.448398 392 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/krb5kdc/test-user.keytab.
Sep 25 04:39:46 dist-test-slave-dc95 krb5kdc[10602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:39:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:39:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 04:39:46.575968 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41579
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41579
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:39:46.686420 10615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:39:46.686731 10615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:39:46.686774 10615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:39:46.690924 10615 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:39:46.691004 10615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:39:46.691023 10615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:39:46.691039 10615 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:39:46.691053 10615 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:39:46.696820 10615 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41579
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:39:46.697943 10615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:39:46.698961 10615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:39:46.707647 10620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:46.707648 10621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:46.707691 10615 server_base.cc:1034] running on GCE node
W20260925 04:39:46.707648 10623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:46.708508 10615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:39:46.709667 10615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:39:46.710844 10615 hybrid_clock.cc:648] HybridClock initialized: now 1790311186710816 us; error 42 us; skew 500 ppm
Sep 25 04:39:46 dist-test-slave-dc95 krb5kdc[10602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311186, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:39:46.713897 10615 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 04:39:46.714916 10615 webserver.cc:460] Webserver started at http://127.0.98.62:36669/ using document root <none> and password file <none>
I20260925 04:39:46.715474 10615 fs_manager.cc:359] Metadata directory not provided
I20260925 04:39:46.715545 10615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:39:46.715788 10615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:39:46.717947 10615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/minicluster-data/master-0/data/instance:
uuid: "99f6d4e61c9b4178867feaa74096983c"
format_stamp: "Formatted at 2026-09-25 04:39:46 on dist-test-slave-dc95"
server_key: "34f9a39f779ce3711d1a3a57a33a9964"
server_key_iv: "26f143e3fa455b2b6e8231d93a4b5d57"
server_key_version: "encryptionkey@0"
I20260925 04:39:46.718513 10615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/minicluster-data/master-0/wal/instance:
uuid: "99f6d4e61c9b4178867feaa74096983c"
format_stamp: "Formatted at 2026-09-25 04:39:46 on dist-test-slave-dc95"
server_key: "34f9a39f779ce3711d1a3a57a33a9964"
server_key_iv: "26f143e3fa455b2b6e8231d93a4b5d57"
server_key_version: "encryptionkey@0"
I20260925 04:39:46.722869 10615 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 04:39:46.725560 10630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:46.726969 10615 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:39:46.727128 10615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/minicluster-data/master-0/wal
uuid: "99f6d4e61c9b4178867feaa74096983c"
format_stamp: "Formatted at 2026-09-25 04:39:46 on dist-test-slave-dc95"
server_key: "34f9a39f779ce3711d1a3a57a33a9964"
server_key_iv: "26f143e3fa455b2b6e8231d93a4b5d57"
server_key_version: "encryptionkey@0"
I20260925 04:39:46.727237 10615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:39:46.738829 10615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:39:46.742333 10615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:39:46.742534 10615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:39:46.749593 10615 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41579
I20260925 04:39:46.749919 10682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41579 every 8 connection(s)
I20260925 04:39:46.750583 10615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
I20260925 04:39:46.752384 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 10615
I20260925 04:39:46.752496 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/minicluster-data/master-0/wal/instance
I20260925 04:39:46.752763 392 external_mini_cluster.cc:1428] Setting key 1ed389b55db6c95b3730107d8910b34e
I20260925 04:39:46.753650 10683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:39:46.760440 10683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99f6d4e61c9b4178867feaa74096983c: Bootstrap starting.
Sep 25 04:39:46 dist-test-slave-dc95 krb5kdc[10602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:39:46.763377 10683 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99f6d4e61c9b4178867feaa74096983c: Neither blocks nor log segments found. Creating new log.
I20260925 04:39:46.764360 10683 log.cc:824] T 00000000000000000000000000000000 P 99f6d4e61c9b4178867feaa74096983c: Log is configured to *not* fsync() on all Append() calls
I20260925 04:39:46.766940 10683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99f6d4e61c9b4178867feaa74096983c: No bootstrap required, opened a new log
I20260925 04:39:46.768007 10686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:46.754112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50822 (local address 127.0.98.62:41579)
0925 04:39:46.754502 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:46.754518 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:46.754554 (+ 36us) server_negotiation.cc:407] Connection header received
0925 04:39:46.755203 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:46.755222 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:46.755584 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:46.755994 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:46.756915 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:46.757966 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:46.759175 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:46.759748 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:46.759879 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:46.762644 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:46.762651 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:46.762668 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:46.762710 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:46.764683 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:46.765204 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:46.765210 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:46.765215 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:46.765344 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:46.765685 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:46.765688 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:46.765690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:46.766218 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:46.766394 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:46.766687 (+ 293us) server_negotiation.cc:299] Negotiation successful
0925 04:39:46.766996 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":105,"threads_started":1}
I20260925 04:39:46.771106 10683 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99f6d4e61c9b4178867feaa74096983c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99f6d4e61c9b4178867feaa74096983c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41579 } }
I20260925 04:39:46.771348 10683 raft_consensus.cc:374] T 00000000000000000000000000000000 P 99f6d4e61c9b4178867feaa74096983c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:39:46.771384 10683 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99f6d4e61c9b4178867feaa74096983c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99f6d4e61c9b4178867feaa74096983c, State: Initialized, Role: FOLLOWER
I20260925 04:39:46.771793 10683 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99f6d4e61c9b4178867feaa74096983c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99f6d4e61c9b4178867feaa74096983c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41579 } }
I20260925 04:39:46.771924 10683 raft_consensus.cc:388] T 00000000000000000000000000000000 P 99f6d4e61c9b4178867feaa74096983c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:39:46.771962 10683 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99f6d4e61c9b4178867feaa74096983c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:39:46.772040 10683 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99f6d4e61c9b4178867feaa74096983c [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:39:46.772959 10683 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99f6d4e61c9b4178867feaa74096983c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99f6d4e61c9b4178867feaa74096983c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41579 } }
I20260925 04:39:46.773278 10683 leader_election.cc:302] T 00000000000000000000000000000000 P 99f6d4e61c9b4178867feaa74096983c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99f6d4e61c9b4178867feaa74096983c; no voters:
I20260925 04:39:46.773561 10683 leader_election.cc:288] T 00000000000000000000000000000000 P 99f6d4e61c9b4178867feaa74096983c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:39:46.773701 10688 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99f6d4e61c9b4178867feaa74096983c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:39:46.773927 10688 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99f6d4e61c9b4178867feaa74096983c [term 1 LEADER]: Becoming Leader. State: Replica: 99f6d4e61c9b4178867feaa74096983c, State: Running, Role: LEADER
I20260925 04:39:46.774318 10688 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99f6d4e61c9b4178867feaa74096983c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99f6d4e61c9b4178867feaa74096983c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41579 } }
I20260925 04:39:46.774814 10683 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99f6d4e61c9b4178867feaa74096983c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:39:46.775969 10689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99f6d4e61c9b4178867feaa74096983c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "99f6d4e61c9b4178867feaa74096983c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99f6d4e61c9b4178867feaa74096983c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41579 } } }
I20260925 04:39:46.776103 10689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99f6d4e61c9b4178867feaa74096983c [sys.catalog]: This master's current role is: LEADER
I20260925 04:39:46.776326 10690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99f6d4e61c9b4178867feaa74096983c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99f6d4e61c9b4178867feaa74096983c. Latest consensus state: current_term: 1 leader_uuid: "99f6d4e61c9b4178867feaa74096983c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99f6d4e61c9b4178867feaa74096983c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41579 } } }
I20260925 04:39:46.776490 10690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99f6d4e61c9b4178867feaa74096983c [sys.catalog]: This master's current role is: LEADER
I20260925 04:39:46.776585 10698 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:39:46.779346 10698 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:39:46.785511 10698 catalog_manager.cc:1347] Generated new cluster ID: 898e1442f95848569ec44f6b78a3fe12
I20260925 04:39:46.785605 10698 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:39:46.804311 10698 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:39:46.805680 10698 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:39:46.813421 10698 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 99f6d4e61c9b4178867feaa74096983c: Generated new TSK 0
I20260925 04:39:46.814213 10698 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 04:39:46.851752 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41579
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:39:46.970048 10709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:39:46.970337 10709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:39:46.970402 10709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:39:46.974597 10709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:39:46.974673 10709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:39:46.974759 10709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 04:39:46.980371 10709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:39:46.981480 10709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:39:46.982547 10709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:39:46.992439 10714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:46.992487 10715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:46.992486 10717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:46.992527 10709 server_base.cc:1034] running on GCE node
I20260925 04:39:46.993080 10709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:39:46.993770 10709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:39:46.994968 10709 hybrid_clock.cc:648] HybridClock initialized: now 1790311186994935 us; error 44 us; skew 500 ppm
Sep 25 04:39:46 dist-test-slave-dc95 krb5kdc[10602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311186, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:39:46.997727 10709 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 04:39:46.998643 10709 webserver.cc:460] Webserver started at http://127.0.98.1:42697/ using document root <none> and password file <none>
I20260925 04:39:46.999198 10709 fs_manager.cc:359] Metadata directory not provided
I20260925 04:39:46.999262 10709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:39:46.999480 10709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:39:47.001662 10709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/minicluster-data/ts-0/data/instance:
uuid: "afa49094a6cf448f9e772252f2842fd5"
format_stamp: "Formatted at 2026-09-25 04:39:46 on dist-test-slave-dc95"
server_key: "79638aa4682ceb51c547c940c97003f8"
server_key_iv: "fd32b9808b86c268f721cf810253a9a8"
server_key_version: "encryptionkey@0"
I20260925 04:39:47.002261 10709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/minicluster-data/ts-0/wal/instance:
uuid: "afa49094a6cf448f9e772252f2842fd5"
format_stamp: "Formatted at 2026-09-25 04:39:46 on dist-test-slave-dc95"
server_key: "79638aa4682ceb51c547c940c97003f8"
server_key_iv: "fd32b9808b86c268f721cf810253a9a8"
server_key_version: "encryptionkey@0"
I20260925 04:39:47.006191 10709 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 04:39:47.008757 10724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:47.010004 10709 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:39:47.010123 10709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/minicluster-data/ts-0/wal
uuid: "afa49094a6cf448f9e772252f2842fd5"
format_stamp: "Formatted at 2026-09-25 04:39:46 on dist-test-slave-dc95"
server_key: "79638aa4682ceb51c547c940c97003f8"
server_key_iv: "fd32b9808b86c268f721cf810253a9a8"
server_key_version: "encryptionkey@0"
I20260925 04:39:47.010231 10709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:39:47.037554 10709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:39:47.041064 10709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:39:47.041260 10709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:39:47.041880 10709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:39:47.042778 10709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:39:47.042829 10709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:47.042884 10709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:39:47.042909 10709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:47.051424 10709 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34627
I20260925 04:39:47.051448 10837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34627 every 8 connection(s)
I20260925 04:39:47.052379 10709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
I20260925 04:39:47.059016 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 10709
I20260925 04:39:47.059149 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/minicluster-data/ts-0/wal/instance
I20260925 04:39:47.059454 392 external_mini_cluster.cc:1428] Setting key 5349a08e4206c17bef6de36ae35a29d2
Sep 25 04:39:47 dist-test-slave-dc95 krb5kdc[10602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311186, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:39:47.067584 10686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:47.054176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58585 (local address 127.0.98.62:41579)
0925 04:39:47.054299 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:47.054303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:47.055625 (+ 1322us) server_negotiation.cc:407] Connection header received
0925 04:39:47.056545 (+ 920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:47.056573 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:47.056638 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:47.056719 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:47.058013 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:47.058611 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:47.060191 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:47.060423 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:47.060515 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:47.063368 (+ 2853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:47.063373 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:47.063376 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:47.063410 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:47.065270 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:47.065855 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:47.065859 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:47.065861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:47.065958 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:47.066299 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:47.066303 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:47.066305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:47.066562 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:47.066671 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:47.067246 (+ 575us) server_negotiation.cc:299] Negotiation successful
0925 04:39:47.067388 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 04:39:47.068432 10840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:47.054556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41579 (local address 127.0.98.1:58585)
0925 04:39:47.055441 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:47.055485 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:47.056320 (+ 835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:47.056929 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:47.056937 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:47.057353 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:47.057852 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:47.057868 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:47.058749 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:47.058753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:47.060068 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:47.060079 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:47.060572 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:47.060576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:47.060743 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:47.061324 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:47.061346 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:47.063165 (+ 1819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:39:47.065419 (+ 2254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:47.065426 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:47.065442 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:47.065741 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:47.066058 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:47.066061 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:47.066062 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:47.066210 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:47.066674 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:39:47.066680 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:39:47.066945 (+ 265us) client_negotiation.cc:769] Sending connection context
0925 04:39:47.067162 (+ 217us) client_negotiation.cc:240] Negotiation successful
0925 04:39:47.067443 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":539,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 04:39:47.069828 10838 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41579
I20260925 04:39:47.070135 10838 heartbeater.cc:464] Registering TS with master...
I20260925 04:39:47.070744 10838 heartbeater.cc:511] Master 127.0.98.62:41579 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260925 04:39:47.072598 10647 ts_manager.cc:194] Registered new tserver with Master: afa49094a6cf448f9e772252f2842fd5 (127.0.98.1:34627)
I20260925 04:39:47.073858 10647 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58585
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 04:39:47.089048 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41579
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:39:47.200358 10843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:39:47.200666 10843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:39:47.200728 10843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:39:47.204957 10843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:39:47.205031 10843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:39:47.205124 10843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 04:39:47.210675 10843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:39:47.211872 10843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:39:47.212918 10843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:39:47.222882 10848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:47.222937 10849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:47.222970 10843 server_base.cc:1034] running on GCE node
W20260925 04:39:47.222882 10851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:47.223712 10843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:39:47.224386 10843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:39:47.225561 10843 hybrid_clock.cc:648] HybridClock initialized: now 1790311187225533 us; error 47 us; skew 500 ppm
Sep 25 04:39:47 dist-test-slave-dc95 krb5kdc[10602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311187, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:39:47.228291 10843 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 04:39:47.229213 10843 webserver.cc:460] Webserver started at http://127.0.98.2:38769/ using document root <none> and password file <none>
I20260925 04:39:47.229760 10843 fs_manager.cc:359] Metadata directory not provided
I20260925 04:39:47.229827 10843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:39:47.230042 10843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:39:47.232282 10843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/minicluster-data/ts-1/data/instance:
uuid: "e6bf22d4d9ff44ed9113d3f110aebfbe"
format_stamp: "Formatted at 2026-09-25 04:39:47 on dist-test-slave-dc95"
server_key: "d6ac1a7aafe91321331166df63c1e5e5"
server_key_iv: "b2be719ca033716e961fdad966de55b9"
server_key_version: "encryptionkey@0"
I20260925 04:39:47.232829 10843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e6bf22d4d9ff44ed9113d3f110aebfbe"
format_stamp: "Formatted at 2026-09-25 04:39:47 on dist-test-slave-dc95"
server_key: "d6ac1a7aafe91321331166df63c1e5e5"
server_key_iv: "b2be719ca033716e961fdad966de55b9"
server_key_version: "encryptionkey@0"
I20260925 04:39:47.236846 10843 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:39:47.239642 10858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:47.241191 10843 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:39:47.241341 10843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/minicluster-data/ts-1/wal
uuid: "e6bf22d4d9ff44ed9113d3f110aebfbe"
format_stamp: "Formatted at 2026-09-25 04:39:47 on dist-test-slave-dc95"
server_key: "d6ac1a7aafe91321331166df63c1e5e5"
server_key_iv: "b2be719ca033716e961fdad966de55b9"
server_key_version: "encryptionkey@0"
I20260925 04:39:47.241467 10843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:39:47.269440 10843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:39:47.272866 10843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:39:47.273072 10843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:39:47.273756 10843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:39:47.274755 10843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:39:47.274807 10843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:47.274865 10843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:39:47.274892 10843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:47.284053 10843 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41449
I20260925 04:39:47.284067 10971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41449 every 8 connection(s)
I20260925 04:39:47.285084 10843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
I20260925 04:39:47.285987 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 10843
I20260925 04:39:47.286085 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/minicluster-data/ts-1/wal/instance
I20260925 04:39:47.286355 392 external_mini_cluster.cc:1428] Setting key fc86305085c3390b193b4cf549ebcfcf
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Sep 25 04:39:47 dist-test-slave-dc95 krb5kdc[10602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311187, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:39:47.304796 10686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:47.288084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36015 (local address 127.0.98.62:41579)
0925 04:39:47.288237 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:47.288241 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:47.289296 (+ 1055us) server_negotiation.cc:407] Connection header received
0925 04:39:47.290910 (+ 1614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:47.290914 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:47.290979 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:47.291074 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:47.292874 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:47.293406 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:47.295248 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:47.295486 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:47.295644 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:47.300896 (+ 5252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:47.300900 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:47.300902 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:47.300934 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:47.302381 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:47.303012 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:47.303015 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:47.303016 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:47.303167 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:47.303531 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:47.303534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:47.303536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:47.303785 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:47.303882 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:47.304487 (+ 605us) server_negotiation.cc:299] Negotiation successful
0925 04:39:47.304628 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 04:39:47.305668 10974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:47.288537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41579 (local address 127.0.98.2:36015)
0925 04:39:47.289094 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:47.289162 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:47.290627 (+ 1465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:47.291300 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:47.291310 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:47.291903 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:47.292663 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:47.292689 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:47.293565 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:47.293569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:47.295079 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:47.295094 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:47.295709 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:47.295714 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:47.295935 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:47.296577 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:47.296599 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:47.300644 (+ 4045us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:39:47.302543 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:47.302549 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:47.302566 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:47.302879 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:47.303266 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:47.303269 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:47.303270 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:47.303431 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:47.303892 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:39:47.303901 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:39:47.304215 (+ 314us) client_negotiation.cc:769] Sending connection context
0925 04:39:47.304406 (+ 191us) client_negotiation.cc:240] Negotiation successful
0925 04:39:47.304780 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":133,"threads_started":1}
I20260925 04:39:47.306994 10972 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41579
I20260925 04:39:47.307291 10972 heartbeater.cc:464] Registering TS with master...
I20260925 04:39:47.307973 10972 heartbeater.cc:511] Master 127.0.98.62:41579 requested a full tablet report, sending...
I20260925 04:39:47.309093 10647 ts_manager.cc:194] Registered new tserver with Master: e6bf22d4d9ff44ed9113d3f110aebfbe (127.0.98.2:41449)
I20260925 04:39:47.309749 10647 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36015
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 04:39:47.316854 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41579
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:39:47.432296 10977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:39:47.432593 10977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:39:47.432638 10977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:39:47.436822 10977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:39:47.436902 10977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:39:47.436987 10977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 04:39:47.443065 10977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:39:47.444284 10977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:39:47.445384 10977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:39:47.455329 10983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:39:47.455341 10982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:47.455448 10977 server_base.cc:1034] running on GCE node
W20260925 04:39:47.455355 10985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:39:47.456089 10977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:39:47.456765 10977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:39:47.457921 10977 hybrid_clock.cc:648] HybridClock initialized: now 1790311187457904 us; error 37 us; skew 500 ppm
Sep 25 04:39:47 dist-test-slave-dc95 krb5kdc[10602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311187, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:39:47.460702 10977 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 04:39:47.461664 10977 webserver.cc:460] Webserver started at http://127.0.98.3:37257/ using document root <none> and password file <none>
I20260925 04:39:47.462229 10977 fs_manager.cc:359] Metadata directory not provided
I20260925 04:39:47.462297 10977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:39:47.462491 10977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:39:47.464627 10977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/minicluster-data/ts-2/data/instance:
uuid: "b9e2916ca32247d6928627bfafdf7e29"
format_stamp: "Formatted at 2026-09-25 04:39:47 on dist-test-slave-dc95"
server_key: "cc8883bc1ca65a45ad65f8aad0f31769"
server_key_iv: "6d8fac78491bef656496d35196e235de"
server_key_version: "encryptionkey@0"
I20260925 04:39:47.465224 10977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b9e2916ca32247d6928627bfafdf7e29"
format_stamp: "Formatted at 2026-09-25 04:39:47 on dist-test-slave-dc95"
server_key: "cc8883bc1ca65a45ad65f8aad0f31769"
server_key_iv: "6d8fac78491bef656496d35196e235de"
server_key_version: "encryptionkey@0"
I20260925 04:39:47.469208 10977 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:39:47.471793 10992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:47.473058 10977 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:39:47.473213 10977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/minicluster-data/ts-2/wal
uuid: "b9e2916ca32247d6928627bfafdf7e29"
format_stamp: "Formatted at 2026-09-25 04:39:47 on dist-test-slave-dc95"
server_key: "cc8883bc1ca65a45ad65f8aad0f31769"
server_key_iv: "6d8fac78491bef656496d35196e235de"
server_key_version: "encryptionkey@0"
I20260925 04:39:47.473325 10977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:39:47.495034 10977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:39:47.498801 10977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:39:47.499043 10977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:39:47.499940 10977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:39:47.501245 10977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:39:47.501350 10977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:47.501425 10977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:39:47.501456 10977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:39:47.510854 10977 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40813
I20260925 04:39:47.510879 11105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40813 every 8 connection(s)
I20260925 04:39:47.511868 10977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
I20260925 04:39:47.513914 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 10977
I20260925 04:39:47.514089 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/minicluster-data/ts-2/wal/instance
I20260925 04:39:47.514389 392 external_mini_cluster.cc:1428] Setting key e6a2a996368c706f874fd280fad93d43
Sep 25 04:39:47 dist-test-slave-dc95 krb5kdc[10602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311187, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:39:47.526793 10686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:47.513869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33309 (local address 127.0.98.62:41579)
0925 04:39:47.514030 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:47.514035 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:47.515078 (+ 1043us) server_negotiation.cc:407] Connection header received
0925 04:39:47.516702 (+ 1624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:47.516705 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:47.516754 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:47.516826 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:47.518109 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:47.518660 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:47.519936 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:47.520214 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:47.520314 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:47.523015 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:47.523019 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:47.523021 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:47.523052 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:47.524673 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:47.525243 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:47.525247 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:47.525248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:47.525338 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:47.525737 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:47.525740 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:47.525741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:47.525933 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:47.526009 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:47.526553 (+ 544us) server_negotiation.cc:299] Negotiation successful
0925 04:39:47.526668 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 04:39:47.527997 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:47.514344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41579 (local address 127.0.98.3:33309)
0925 04:39:47.514924 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:47.514988 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:47.516478 (+ 1490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:47.517008 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:47.517018 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:47.517390 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:39:47.517919 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:47.517933 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:47.518784 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:47.518788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:47.519808 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:47.519819 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:47.520398 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:47.520401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:47.520563 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:47.521228 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:39:47.521246 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:39:47.522809 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:39:47.524808 (+ 1999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:47.524815 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:47.524830 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:47.525134 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:47.525450 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:39:47.525453 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:39:47.525454 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:39:47.525637 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:39:47.526031 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:39:47.526037 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:39:47.526319 (+ 282us) client_negotiation.cc:769] Sending connection context
0925 04:39:47.526526 (+ 207us) client_negotiation.cc:240] Negotiation successful
0925 04:39:47.526901 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":176,"threads_started":1}
I20260925 04:39:47.529083 11106 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41579
I20260925 04:39:47.529327 11106 heartbeater.cc:464] Registering TS with master...
I20260925 04:39:47.529838 11106 heartbeater.cc:511] Master 127.0.98.62:41579 requested a full tablet report, sending...
I20260925 04:39:47.531028 10647 ts_manager.cc:194] Registered new tserver with Master: b9e2916ca32247d6928627bfafdf7e29 (127.0.98.3:40813)
I20260925 04:39:47.531705 10647 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33309
I20260925 04:39:47.540596 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:39:47.550072 10686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:47.542546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50836 (local address 127.0.98.62:41579)
0925 04:39:47.542712 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:47.542716 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:47.542789 (+ 73us) server_negotiation.cc:407] Connection header received
0925 04:39:47.542890 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:47.542892 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:47.542936 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:47.543000 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:39:47.543721 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:47.544242 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:47.545121 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:47.545403 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:47.545487 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:47.546918 (+ 1431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:39:47.546921 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:39:47.546924 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:39:47.546951 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:39:47.548408 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:47.548819 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:47.548822 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:47.548824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:47.548904 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:39:47.549194 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:39:47.549197 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:39:47.549198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:39:47.549417 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:39:47.549516 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:47.549794 (+ 278us) server_negotiation.cc:299] Negotiation successful
0925 04:39:47.549895 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 04:39:47.553145 10647 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50836:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:39:47.556161 10647 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:39:47.570957 11122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:47.565768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46082 (local address 127.0.98.3:40813)
0925 04:39:47.566243 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:47.566255 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:47.566449 (+ 194us) server_negotiation.cc:407] Connection header received
0925 04:39:47.566570 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:47.566575 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:47.566737 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:47.566875 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:39:47.567290 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:47.568389 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:47.569657 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:47.570385 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:47.570494 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:47.570642 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:47.570713 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 04:39:47.570807 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":119,"threads_started":1}
I20260925 04:39:47.570950 11118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:47.565350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40813 (local address 127.0.0.1:46082)
0925 04:39:47.566370 (+ 1020us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:47.566382 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:47.566470 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:47.567017 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:47.567020 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:47.567036 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:39:47.567141 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:47.567178 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:47.568505 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:47.568509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:47.569549 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:47.569557 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:47.570487 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:47.570489 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:47.570563 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:47.570593 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 04:39:47.570670 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 04:39:47.570788 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":964,"mutex_wait_us":44,"thread_start_us":78,"threads_started":1}
I20260925 04:39:47.571228 11121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:47.565764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40230 (local address 127.0.98.1:34627)
0925 04:39:47.566030 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:47.566044 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:47.566301 (+ 257us) server_negotiation.cc:407] Connection header received
0925 04:39:47.566386 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:47.566392 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:47.566617 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:47.566784 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:39:47.567074 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:47.568285 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:47.569901 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:47.570633 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:47.570731 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:47.570854 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:47.571041 (+ 187us) server_negotiation.cc:299] Negotiation successful
0925 04:39:47.571126 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":72,"threads_started":1}
I20260925 04:39:47.571456 11120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:47.564985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34627 (local address 127.0.0.1:40230)
0925 04:39:47.566125 (+ 1140us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:47.566135 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:47.566233 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:47.566785 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:47.566792 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:47.566814 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:39:47.566961 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:47.566968 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:47.568413 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:47.568416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:47.569791 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:47.569797 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:47.570745 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:47.570748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:47.570832 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:47.570852 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 04:39:47.570914 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 04:39:47.570968 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1087,"mutex_wait_us":66,"spinlock_wait_cycles":11648,"thread_start_us":101,"threads_started":1}
I20260925 04:39:47.572225 11117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:47.564954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41449 (local address 127.0.0.1:40374)
0925 04:39:47.565782 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:47.565835 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:47.565960 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:47.566968 (+ 1008us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:47.566970 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:47.566985 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:39:47.567104 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:47.567111 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:47.568127 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:47.568130 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:47.569785 (+ 1655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:47.569795 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:47.571852 (+ 2057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:47.571855 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:47.571941 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:47.571961 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 04:39:47.572025 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 04:39:47.572089 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"mutex_wait_us":80,"thread_start_us":94,"threads_started":1}
I20260925 04:39:47.572269 11119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:47.565104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40374 (local address 127.0.98.2:41449)
0925 04:39:47.566471 (+ 1367us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:47.566481 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:47.566499 (+ 18us) server_negotiation.cc:407] Connection header received
0925 04:39:47.566562 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:47.566566 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:47.566712 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:47.566825 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:39:47.567222 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:47.568012 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:47.570916 (+ 2904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:47.571725 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:47.571832 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:47.571961 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:47.572048 (+ 87us) server_negotiation.cc:299] Negotiation successful
0925 04:39:47.572135 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1275,"thread_start_us":102,"threads_started":1}
I20260925 04:39:47.574323 11040 tablet_service.cc:1467] Processing CreateTablet for tablet 8aaa8ced336c4d4a84cd3d3330d1fbe4 (DEFAULT_TABLE table=test-table [id=c45fbd31ead048b8adb1399d1f9d023f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:39:47.574360 10772 tablet_service.cc:1467] Processing CreateTablet for tablet 8aaa8ced336c4d4a84cd3d3330d1fbe4 (DEFAULT_TABLE table=test-table [id=c45fbd31ead048b8adb1399d1f9d023f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:39:47.575362 11040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8aaa8ced336c4d4a84cd3d3330d1fbe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:39:47.575374 10772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8aaa8ced336c4d4a84cd3d3330d1fbe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:39:47.575744 10906 tablet_service.cc:1467] Processing CreateTablet for tablet 8aaa8ced336c4d4a84cd3d3330d1fbe4 (DEFAULT_TABLE table=test-table [id=c45fbd31ead048b8adb1399d1f9d023f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:39:47.576819 10906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8aaa8ced336c4d4a84cd3d3330d1fbe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:39:47.581791 11123 tablet_bootstrap.cc:492] T 8aaa8ced336c4d4a84cd3d3330d1fbe4 P b9e2916ca32247d6928627bfafdf7e29: Bootstrap starting.
I20260925 04:39:47.583982 11125 tablet_bootstrap.cc:492] T 8aaa8ced336c4d4a84cd3d3330d1fbe4 P e6bf22d4d9ff44ed9113d3f110aebfbe: Bootstrap starting.
I20260925 04:39:47.584012 11123 tablet_bootstrap.cc:654] T 8aaa8ced336c4d4a84cd3d3330d1fbe4 P b9e2916ca32247d6928627bfafdf7e29: Neither blocks nor log segments found. Creating new log.
I20260925 04:39:47.584563 11124 tablet_bootstrap.cc:492] T 8aaa8ced336c4d4a84cd3d3330d1fbe4 P afa49094a6cf448f9e772252f2842fd5: Bootstrap starting.
I20260925 04:39:47.585613 11123 log.cc:824] T 8aaa8ced336c4d4a84cd3d3330d1fbe4 P b9e2916ca32247d6928627bfafdf7e29: Log is configured to *not* fsync() on all Append() calls
I20260925 04:39:47.586871 11125 tablet_bootstrap.cc:654] T 8aaa8ced336c4d4a84cd3d3330d1fbe4 P e6bf22d4d9ff44ed9113d3f110aebfbe: Neither blocks nor log segments found. Creating new log.
I20260925 04:39:47.586992 11124 tablet_bootstrap.cc:654] T 8aaa8ced336c4d4a84cd3d3330d1fbe4 P afa49094a6cf448f9e772252f2842fd5: Neither blocks nor log segments found. Creating new log.
I20260925 04:39:47.587718 11123 tablet_bootstrap.cc:492] T 8aaa8ced336c4d4a84cd3d3330d1fbe4 P b9e2916ca32247d6928627bfafdf7e29: No bootstrap required, opened a new log
I20260925 04:39:47.587821 11124 log.cc:824] T 8aaa8ced336c4d4a84cd3d3330d1fbe4 P afa49094a6cf448f9e772252f2842fd5: Log is configured to *not* fsync() on all Append() calls
I20260925 04:39:47.587822 11125 log.cc:824] T 8aaa8ced336c4d4a84cd3d3330d1fbe4 P e6bf22d4d9ff44ed9113d3f110aebfbe: Log is configured to *not* fsync() on all Append() calls
I20260925 04:39:47.587916 11123 ts_tablet_manager.cc:1397] T 8aaa8ced336c4d4a84cd3d3330d1fbe4 P b9e2916ca32247d6928627bfafdf7e29: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260925 04:39:47.589897 11124 tablet_bootstrap.cc:492] T 8aaa8ced336c4d4a84cd3d3330d1fbe4 P afa49094a6cf448f9e772252f2842fd5: No bootstrap required, opened a new log
I20260925 04:39:47.590108 11124 ts_tablet_manager.cc:1397] T 8aaa8ced336c4d4a84cd3d3330d1fbe4 P afa49094a6cf448f9e772252f2842fd5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 04:39:47.590176 11125 tablet_bootstrap.cc:492] T 8aaa8ced336c4d4a84cd3d3330d1fbe4 P e6bf22d4d9ff44ed9113d3f110aebfbe: No bootstrap required, opened a new log
I20260925 04:39:47.590404 11125 ts_tablet_manager.cc:1397] T 8aaa8ced336c4d4a84cd3d3330d1fbe4 P e6bf22d4d9ff44ed9113d3f110aebfbe: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260925 04:39:47.591429 11123 raft_consensus.cc:348] T 8aaa8ced336c4d4a84cd3d3330d1fbe4 P b9e2916ca32247d6928627bfafdf7e29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6bf22d4d9ff44ed9113d3f110aebfbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41449 } } peers { permanent_uuid: "afa49094a6cf448f9e772252f2842fd5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34627 } } peers { permanent_uuid: "b9e2916ca32247d6928627bfafdf7e29" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40813 } }
I20260925 04:39:47.591908 11123 raft_consensus.cc:374] T 8aaa8ced336c4d4a84cd3d3330d1fbe4 P b9e2916ca32247d6928627bfafdf7e29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:39:47.591971 11123 raft_consensus.cc:729] T 8aaa8ced336c4d4a84cd3d3330d1fbe4 P b9e2916ca32247d6928627bfafdf7e29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9e2916ca32247d6928627bfafdf7e29, State: Initialized, Role: FOLLOWER
I20260925 04:39:47.592449 11123 consensus_queue.cc:260] T 8aaa8ced336c4d4a84cd3d3330d1fbe4 P b9e2916ca32247d6928627bfafdf7e29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6bf22d4d9ff44ed9113d3f110aebfbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41449 } } peers { permanent_uuid: "afa49094a6cf448f9e772252f2842fd5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34627 } } peers { permanent_uuid: "b9e2916ca32247d6928627bfafdf7e29" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40813 } }
I20260925 04:39:47.593122 11106 heartbeater.cc:503] Master 127.0.98.62:41579 was elected leader, sending a full tablet report...
I20260925 04:39:47.593528 11123 ts_tablet_manager.cc:1428] T 8aaa8ced336c4d4a84cd3d3330d1fbe4 P b9e2916ca32247d6928627bfafdf7e29: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260925 04:39:47.594462 11125 raft_consensus.cc:348] T 8aaa8ced336c4d4a84cd3d3330d1fbe4 P e6bf22d4d9ff44ed9113d3f110aebfbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6bf22d4d9ff44ed9113d3f110aebfbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41449 } } peers { permanent_uuid: "afa49094a6cf448f9e772252f2842fd5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34627 } } peers { permanent_uuid: "b9e2916ca32247d6928627bfafdf7e29" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40813 } }
I20260925 04:39:47.594784 11125 raft_consensus.cc:374] T 8aaa8ced336c4d4a84cd3d3330d1fbe4 P e6bf22d4d9ff44ed9113d3f110aebfbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:39:47.594887 11125 raft_consensus.cc:729] T 8aaa8ced336c4d4a84cd3d3330d1fbe4 P e6bf22d4d9ff44ed9113d3f110aebfbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6bf22d4d9ff44ed9113d3f110aebfbe, State: Initialized, Role: FOLLOWER
I20260925 04:39:47.594841 11124 raft_consensus.cc:348] T 8aaa8ced336c4d4a84cd3d3330d1fbe4 P afa49094a6cf448f9e772252f2842fd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6bf22d4d9ff44ed9113d3f110aebfbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41449 } } peers { permanent_uuid: "afa49094a6cf448f9e772252f2842fd5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34627 } } peers { permanent_uuid: "b9e2916ca32247d6928627bfafdf7e29" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40813 } }
I20260925 04:39:47.595108 11124 raft_consensus.cc:374] T 8aaa8ced336c4d4a84cd3d3330d1fbe4 P afa49094a6cf448f9e772252f2842fd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:39:47.595306 11124 raft_consensus.cc:729] T 8aaa8ced336c4d4a84cd3d3330d1fbe4 P afa49094a6cf448f9e772252f2842fd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afa49094a6cf448f9e772252f2842fd5, State: Initialized, Role: FOLLOWER
I20260925 04:39:47.595402 11125 consensus_queue.cc:260] T 8aaa8ced336c4d4a84cd3d3330d1fbe4 P e6bf22d4d9ff44ed9113d3f110aebfbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6bf22d4d9ff44ed9113d3f110aebfbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41449 } } peers { permanent_uuid: "afa49094a6cf448f9e772252f2842fd5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34627 } } peers { permanent_uuid: "b9e2916ca32247d6928627bfafdf7e29" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40813 } }
I20260925 04:39:47.595809 11124 consensus_queue.cc:260] T 8aaa8ced336c4d4a84cd3d3330d1fbe4 P afa49094a6cf448f9e772252f2842fd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6bf22d4d9ff44ed9113d3f110aebfbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41449 } } peers { permanent_uuid: "afa49094a6cf448f9e772252f2842fd5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34627 } } peers { permanent_uuid: "b9e2916ca32247d6928627bfafdf7e29" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40813 } }
I20260925 04:39:47.596318 10972 heartbeater.cc:503] Master 127.0.98.62:41579 was elected leader, sending a full tablet report...
I20260925 04:39:47.596505 10838 heartbeater.cc:503] Master 127.0.98.62:41579 was elected leader, sending a full tablet report...
I20260925 04:39:47.596570 11125 ts_tablet_manager.cc:1428] T 8aaa8ced336c4d4a84cd3d3330d1fbe4 P e6bf22d4d9ff44ed9113d3f110aebfbe: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260925 04:39:47.596762 11124 ts_tablet_manager.cc:1428] T 8aaa8ced336c4d4a84cd3d3330d1fbe4 P afa49094a6cf448f9e772252f2842fd5: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260925 04:39:47.763025 11107 tablet.cc:2369] T 8aaa8ced336c4d4a84cd3d3330d1fbe4 P b9e2916ca32247d6928627bfafdf7e29: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 04:39:47.786317 10973 tablet.cc:2369] T 8aaa8ced336c4d4a84cd3d3330d1fbe4 P e6bf22d4d9ff44ed9113d3f110aebfbe: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 04:39:47.803803 10839 tablet.cc:2369] T 8aaa8ced336c4d4a84cd3d3330d1fbe4 P afa49094a6cf448f9e772252f2842fd5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:39:47.808274 11130 raft_consensus.cc:482] T 8aaa8ced336c4d4a84cd3d3330d1fbe4 P e6bf22d4d9ff44ed9113d3f110aebfbe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:39:47.808497 11130 raft_consensus.cc:504] T 8aaa8ced336c4d4a84cd3d3330d1fbe4 P e6bf22d4d9ff44ed9113d3f110aebfbe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6bf22d4d9ff44ed9113d3f110aebfbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41449 } } peers { permanent_uuid: "afa49094a6cf448f9e772252f2842fd5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34627 } } peers { permanent_uuid: "b9e2916ca32247d6928627bfafdf7e29" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40813 } }
I20260925 04:39:47.809669 11130 leader_election.cc:288] T 8aaa8ced336c4d4a84cd3d3330d1fbe4 P e6bf22d4d9ff44ed9113d3f110aebfbe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers afa49094a6cf448f9e772252f2842fd5 (127.0.98.1:34627), b9e2916ca32247d6928627bfafdf7e29 (127.0.98.3:40813)
I20260925 04:39:47.814977 11121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:47.810154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34821 (local address 127.0.98.1:34627)
0925 04:39:47.810274 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:47.810279 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:47.810393 (+ 114us) server_negotiation.cc:407] Connection header received
0925 04:39:47.810563 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:47.810565 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:47.810628 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:47.810713 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:39:47.811057 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:47.811867 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:47.813680 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:47.814332 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:47.814471 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:47.814538 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:47.814762 (+ 224us) server_negotiation.cc:299] Negotiation successful
0925 04:39:47.814851 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 04:39:47.815026 11132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:47.809961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40813 (local address 127.0.98.2:47575)
0925 04:39:47.810296 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:47.810311 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:47.810450 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:47.810731 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:47.810734 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:47.810766 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:39:47.810951 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:47.810957 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:47.812052 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:47.812055 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:47.813548 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:47.813554 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:47.814605 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:47.814609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:47.814724 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:47.814748 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 04:39:47.814825 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 04:39:47.814895 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":199,"spinlock_wait_cycles":15360,"thread_start_us":76,"threads_started":1}
I20260925 04:39:47.814981 11133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:47.809961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34627 (local address 127.0.98.2:34821)
0925 04:39:47.810296 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:47.810311 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:39:47.810451 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:39:47.810731 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:39:47.810734 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:39:47.810764 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:39:47.810962 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:47.810967 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:47.812012 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:47.812015 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:47.813523 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:39:47.813538 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:39:47.814521 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:39:47.814524 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:39:47.814629 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:47.814654 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 04:39:47.814727 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 04:39:47.814794 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":105,"spinlock_wait_cycles":31488,"thread_start_us":73,"threads_started":1}
I20260925 04:39:47.815124 11122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:47.810096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47575 (local address 127.0.98.3:40813)
0925 04:39:47.810271 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:47.810276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:47.810393 (+ 117us) server_negotiation.cc:407] Connection header received
0925 04:39:47.810557 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:47.810560 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:47.810615 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:47.810713 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:39:47.811057 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:47.811867 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:47.813681 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:47.814375 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:47.814470 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:47.814538 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:47.814906 (+ 368us) server_negotiation.cc:299] Negotiation successful
0925 04:39:47.814992 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260925 04:39:47.815889 10792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aaa8ced336c4d4a84cd3d3330d1fbe4" candidate_uuid: "e6bf22d4d9ff44ed9113d3f110aebfbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afa49094a6cf448f9e772252f2842fd5" is_pre_election: true
I20260925 04:39:47.815891 11060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aaa8ced336c4d4a84cd3d3330d1fbe4" candidate_uuid: "e6bf22d4d9ff44ed9113d3f110aebfbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9e2916ca32247d6928627bfafdf7e29" is_pre_election: true
I20260925 04:39:47.816174 10792 raft_consensus.cc:2454] T 8aaa8ced336c4d4a84cd3d3330d1fbe4 P afa49094a6cf448f9e772252f2842fd5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6bf22d4d9ff44ed9113d3f110aebfbe in term 0.
I20260925 04:39:47.816175 11060 raft_consensus.cc:2454] T 8aaa8ced336c4d4a84cd3d3330d1fbe4 P b9e2916ca32247d6928627bfafdf7e29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6bf22d4d9ff44ed9113d3f110aebfbe in term 0.
I20260925 04:39:47.816632 10862 leader_election.cc:302] T 8aaa8ced336c4d4a84cd3d3330d1fbe4 P e6bf22d4d9ff44ed9113d3f110aebfbe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9e2916ca32247d6928627bfafdf7e29, e6bf22d4d9ff44ed9113d3f110aebfbe; no voters:
I20260925 04:39:47.816920 11130 raft_consensus.cc:2781] T 8aaa8ced336c4d4a84cd3d3330d1fbe4 P e6bf22d4d9ff44ed9113d3f110aebfbe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:39:47.817015 11130 raft_consensus.cc:482] T 8aaa8ced336c4d4a84cd3d3330d1fbe4 P e6bf22d4d9ff44ed9113d3f110aebfbe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:39:47.817049 11130 raft_consensus.cc:3037] T 8aaa8ced336c4d4a84cd3d3330d1fbe4 P e6bf22d4d9ff44ed9113d3f110aebfbe [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:39:47.818207 11130 raft_consensus.cc:504] T 8aaa8ced336c4d4a84cd3d3330d1fbe4 P e6bf22d4d9ff44ed9113d3f110aebfbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6bf22d4d9ff44ed9113d3f110aebfbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41449 } } peers { permanent_uuid: "afa49094a6cf448f9e772252f2842fd5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34627 } } peers { permanent_uuid: "b9e2916ca32247d6928627bfafdf7e29" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40813 } }
I20260925 04:39:47.818580 11130 leader_election.cc:288] T 8aaa8ced336c4d4a84cd3d3330d1fbe4 P e6bf22d4d9ff44ed9113d3f110aebfbe [CANDIDATE]: Term 1 election: Requested vote from peers afa49094a6cf448f9e772252f2842fd5 (127.0.98.1:34627), b9e2916ca32247d6928627bfafdf7e29 (127.0.98.3:40813)
I20260925 04:39:47.818936 10792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aaa8ced336c4d4a84cd3d3330d1fbe4" candidate_uuid: "e6bf22d4d9ff44ed9113d3f110aebfbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afa49094a6cf448f9e772252f2842fd5"
I20260925 04:39:47.818974 11060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aaa8ced336c4d4a84cd3d3330d1fbe4" candidate_uuid: "e6bf22d4d9ff44ed9113d3f110aebfbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9e2916ca32247d6928627bfafdf7e29"
I20260925 04:39:47.819074 10792 raft_consensus.cc:3037] T 8aaa8ced336c4d4a84cd3d3330d1fbe4 P afa49094a6cf448f9e772252f2842fd5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:39:47.819094 11060 raft_consensus.cc:3037] T 8aaa8ced336c4d4a84cd3d3330d1fbe4 P b9e2916ca32247d6928627bfafdf7e29 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:39:47.820156 10792 raft_consensus.cc:2454] T 8aaa8ced336c4d4a84cd3d3330d1fbe4 P afa49094a6cf448f9e772252f2842fd5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6bf22d4d9ff44ed9113d3f110aebfbe in term 1.
I20260925 04:39:47.820248 11060 raft_consensus.cc:2454] T 8aaa8ced336c4d4a84cd3d3330d1fbe4 P b9e2916ca32247d6928627bfafdf7e29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6bf22d4d9ff44ed9113d3f110aebfbe in term 1.
I20260925 04:39:47.820513 10859 leader_election.cc:302] T 8aaa8ced336c4d4a84cd3d3330d1fbe4 P e6bf22d4d9ff44ed9113d3f110aebfbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: afa49094a6cf448f9e772252f2842fd5, e6bf22d4d9ff44ed9113d3f110aebfbe; no voters:
I20260925 04:39:47.820741 11130 raft_consensus.cc:2781] T 8aaa8ced336c4d4a84cd3d3330d1fbe4 P e6bf22d4d9ff44ed9113d3f110aebfbe [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:39:47.820990 11130 raft_consensus.cc:686] T 8aaa8ced336c4d4a84cd3d3330d1fbe4 P e6bf22d4d9ff44ed9113d3f110aebfbe [term 1 LEADER]: Becoming Leader. State: Replica: e6bf22d4d9ff44ed9113d3f110aebfbe, State: Running, Role: LEADER
I20260925 04:39:47.821399 11130 consensus_queue.cc:237] T 8aaa8ced336c4d4a84cd3d3330d1fbe4 P e6bf22d4d9ff44ed9113d3f110aebfbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6bf22d4d9ff44ed9113d3f110aebfbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41449 } } peers { permanent_uuid: "afa49094a6cf448f9e772252f2842fd5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34627 } } peers { permanent_uuid: "b9e2916ca32247d6928627bfafdf7e29" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40813 } }
I20260925 04:39:47.824658 10645 catalog_manager.cc:5495] T 8aaa8ced336c4d4a84cd3d3330d1fbe4 P e6bf22d4d9ff44ed9113d3f110aebfbe reported cstate change: term changed from 0 to 1, leader changed from <none> to e6bf22d4d9ff44ed9113d3f110aebfbe (127.0.98.2). New cstate: current_term: 1 leader_uuid: "e6bf22d4d9ff44ed9113d3f110aebfbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6bf22d4d9ff44ed9113d3f110aebfbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "afa49094a6cf448f9e772252f2842fd5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9e2916ca32247d6928627bfafdf7e29" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40813 } health_report { overall_health: UNKNOWN } } }
I20260925 04:39:47.869269 11119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:47.865693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40388 (local address 127.0.98.2:41449)
0925 04:39:47.865807 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 04:39:47.865811 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:39:47.865825 (+ 14us) server_negotiation.cc:407] Connection header received
0925 04:39:47.865971 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:39:47.865974 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:39:47.866034 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:39:47.866126 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:39:47.866374 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:47.866882 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:47.867828 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:39:47.868052 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:39:47.868141 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:39:47.868452 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:39:47.868883 (+ 431us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:39:47.868990 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 04:39:47.869051 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 04:39:47.869112 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 04:39:47.875373 11060 raft_consensus.cc:1261] T 8aaa8ced336c4d4a84cd3d3330d1fbe4 P b9e2916ca32247d6928627bfafdf7e29 [term 1 FOLLOWER]: Refusing update from remote peer e6bf22d4d9ff44ed9113d3f110aebfbe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:39:47.875373 10792 raft_consensus.cc:1261] T 8aaa8ced336c4d4a84cd3d3330d1fbe4 P afa49094a6cf448f9e772252f2842fd5 [term 1 FOLLOWER]: Refusing update from remote peer e6bf22d4d9ff44ed9113d3f110aebfbe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:39:47.876327 11130 consensus_queue.cc:1035] T 8aaa8ced336c4d4a84cd3d3330d1fbe4 P e6bf22d4d9ff44ed9113d3f110aebfbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "afa49094a6cf448f9e772252f2842fd5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:39:47.876600 11134 consensus_queue.cc:1035] T 8aaa8ced336c4d4a84cd3d3330d1fbe4 P e6bf22d4d9ff44ed9113d3f110aebfbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9e2916ca32247d6928627bfafdf7e29" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:39:47.894083 10647 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:39:47.894299 10647 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:39:47.897106 10647 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 99f6d4e61c9b4178867feaa74096983c: Sending DeleteTablet for 3 replicas of tablet 8aaa8ced336c4d4a84cd3d3330d1fbe4
I20260925 04:39:47.898078 11040 tablet_service.cc:1514] Processing DeleteTablet for tablet 8aaa8ced336c4d4a84cd3d3330d1fbe4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:39:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46082
I20260925 04:39:47.898072 10906 tablet_service.cc:1514] Processing DeleteTablet for tablet 8aaa8ced336c4d4a84cd3d3330d1fbe4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:39:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40374
I20260925 04:39:47.898144 10772 tablet_service.cc:1514] Processing DeleteTablet for tablet 8aaa8ced336c4d4a84cd3d3330d1fbe4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:39:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40230
I20260925 04:39:47.898794 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 10709
I20260925 04:39:47.898813 11147 tablet_replica.cc:330] T 8aaa8ced336c4d4a84cd3d3330d1fbe4 P b9e2916ca32247d6928627bfafdf7e29: stopping tablet replica
I20260925 04:39:47.898633 11146 tablet_replica.cc:330] T 8aaa8ced336c4d4a84cd3d3330d1fbe4 P afa49094a6cf448f9e772252f2842fd5: stopping tablet replica
I20260925 04:39:47.900978 11148 tablet_replica.cc:330] T 8aaa8ced336c4d4a84cd3d3330d1fbe4 P e6bf22d4d9ff44ed9113d3f110aebfbe: stopping tablet replica
I20260925 04:39:47.901324 11148 raft_consensus.cc:2229] T 8aaa8ced336c4d4a84cd3d3330d1fbe4 P e6bf22d4d9ff44ed9113d3f110aebfbe [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:39:47.901724 11148 raft_consensus.cc:2258] T 8aaa8ced336c4d4a84cd3d3330d1fbe4 P e6bf22d4d9ff44ed9113d3f110aebfbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:39:47.902175 11147 raft_consensus.cc:2229] T 8aaa8ced336c4d4a84cd3d3330d1fbe4 P b9e2916ca32247d6928627bfafdf7e29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:39:47.902685 11147 raft_consensus.cc:2258] T 8aaa8ced336c4d4a84cd3d3330d1fbe4 P b9e2916ca32247d6928627bfafdf7e29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:39:47.903591 11148 ts_tablet_manager.cc:1905] T 8aaa8ced336c4d4a84cd3d3330d1fbe4 P e6bf22d4d9ff44ed9113d3f110aebfbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:39:47.904606 11147 ts_tablet_manager.cc:1905] T 8aaa8ced336c4d4a84cd3d3330d1fbe4 P b9e2916ca32247d6928627bfafdf7e29: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 04:39:47.906322 10631 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34627 (error 108)
I20260925 04:39:47.906798 11148 ts_tablet_manager.cc:1918] T 8aaa8ced336c4d4a84cd3d3330d1fbe4 P e6bf22d4d9ff44ed9113d3f110aebfbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:39:47.907096 11148 log.cc:1196] T 8aaa8ced336c4d4a84cd3d3330d1fbe4 P e6bf22d4d9ff44ed9113d3f110aebfbe: Deleting WAL directory at /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/minicluster-data/ts-1/wal/wals/8aaa8ced336c4d4a84cd3d3330d1fbe4
I20260925 04:39:47.907280 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 10843
I20260925 04:39:47.908874 11117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:47.908493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34627 (local address 127.0.0.1:40246)
0925 04:39:47.908694 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:47.908770 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
W20260925 04:39:47.909086 10631 catalog_manager.cc:4584] TS afa49094a6cf448f9e772252f2842fd5 (127.0.98.1:34627): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8aaa8ced336c4d4a84cd3d3330d1fbe4: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34627: connect: Connection refused (error 111)
I20260925 04:39:47.909626 11147 ts_tablet_manager.cc:1918] T 8aaa8ced336c4d4a84cd3d3330d1fbe4 P b9e2916ca32247d6928627bfafdf7e29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:39:47.909722 11147 log.cc:1196] T 8aaa8ced336c4d4a84cd3d3330d1fbe4 P b9e2916ca32247d6928627bfafdf7e29: Deleting WAL directory at /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311123478858-392-0/minicluster-data/ts-2/wal/wals/8aaa8ced336c4d4a84cd3d3330d1fbe4
I20260925 04:39:47.910015 11147 ts_tablet_manager.cc:1939] T 8aaa8ced336c4d4a84cd3d3330d1fbe4 P b9e2916ca32247d6928627bfafdf7e29: Deleting consensus metadata
I20260925 04:39:47.911252 10634 catalog_manager.cc:4841] TS b9e2916ca32247d6928627bfafdf7e29 (127.0.98.3:40813): tablet 8aaa8ced336c4d4a84cd3d3330d1fbe4 (table test-table [id=c45fbd31ead048b8adb1399d1f9d023f]) successfully deleted
I20260925 04:39:47.912845 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 10977
I20260925 04:39:47.914098 11117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:39:47.913043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41449 (local address 127.0.0.1:40392)
0925 04:39:47.913946 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0925 04:39:47.914005 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:41449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":812}
I20260925 04:39:47.917903 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 10615
2026-09-25T04:39:47Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2434 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:39:47 dist-test-slave-dc95 krb5kdc[11152](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:39:47 dist-test-slave-dc95 krb5kdc[11152](info): set up 2 sockets
Sep 25 04:39:47 dist-test-slave-dc95 krb5kdc[11152](info): commencing operation
krb5kdc: starting...
W20260925 04:39:48.884567 392 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/krb5kdc/test-user.keytab.
Sep 25 04:39:48 dist-test-slave-dc95 krb5kdc[11152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:39:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:39:48Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260925 04:39:49.059371 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskilA2AO/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/postgres -l logfile start
2026-09-25 04:39:51.694 UTC [11182] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 04:39:51.694 UTC [11182] LOG: listening on IPv4 address "127.0.98.20", port 50971
2026-09-25 04:39:51.699 UTC [11182] LOG: listening on Unix socket "/tmp/.s.PGSQL.50971"
2026-09-25 04:39:51.727 UTC [11185] LOG: database system was shut down at 2026-09-25 04:39:50 UTC
2026-09-25 04:39:51.731 UTC [11182] LOG: database system is ready to accept connections
I20260925 04:39:52.600274 392 mini_postgres.cc:96] Postgres bound to 50971
2026-09-25 04:39:52.606 UTC [11193] FATAL: database "slave" does not exist
127.0.98.20:50971 - accepting connections
I20260925 04:39:52.607214 392 mini_ranger.cc:161] Starting Ranger...
I20260925 04:39:52.621922 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 04:39:53.149394 392 mini_ranger.cc:87] Created ranger Postgres database
I20260925 04:39:53.149506 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/ranger-admin
2026-09-25 04:39:53,434 [I] DB FLAVOR :POSTGRES
2026-09-25 04:39:53,434 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 04:39:53,434 [I] Checking connection..
2026-09-25 04:39:53,435 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskilA2AO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskilA2AO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 04:39:53,714 [I] Checking connection passed.
2026-09-25 04:39:53,714 [I] --------- Verifying version history table ---------
2026-09-25 04:39:53,714 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskilA2AO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskilA2AO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 04:39:53,956 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 04:39:53,957 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskilA2AO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskilA2AO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 04:39:54,226 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 04:39:54,226 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 04:39:54,227 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskilA2AO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskilA2AO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50971/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskilA2AO/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 04:39:54.479 UTC [11257] WARNING: there is no transaction in progress
2026-09-25 04:39:54,485 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 04:39:54,486 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskilA2AO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskilA2AO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 04:39:54,726 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 04:39:54,726 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 04:39:54,726 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskilA2AO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskilA2AO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 04:39:54,962 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskilA2AO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskilA2AO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 04:39:55,191 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskilA2AO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskilA2AO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-dc95.c.gcp-upstream.internal','N') ;"
2026-09-25 04:39:55,446 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskilA2AO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskilA2AO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 04:39:55,692 [I] Table x_portal_user does not exist in database ranger
2026-09-25 04:39:55,693 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskilA2AO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskilA2AO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 04:39:55,955 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 04:39:55,956 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskilA2AO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskilA2AO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 04:39:56,222 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskilA2AO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskilA2AO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 04:39:56,458 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 04:39:56,459 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskilA2AO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskilA2AO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50971/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskilA2AO/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 04:39:57.494 UTC [11365] WARNING: there is no transaction in progress
2026-09-25 04:39:57.512 UTC [11365] WARNING: there is no transaction in progress
2026-09-25 04:39:57.529 UTC [11365] WARNING: there is no transaction in progress
2026-09-25 04:39:57.546 UTC [11365] WARNING: there is no transaction in progress
2026-09-25 04:39:57.562 UTC [11365] WARNING: there is no transaction in progress
2026-09-25 04:39:57.581 UTC [11365] WARNING: there is no transaction in progress
2026-09-25 04:39:57.639 UTC [11365] WARNING: there is no transaction in progress
2026-09-25 04:39:57.648 UTC [11365] WARNING: there is no transaction in progress
2026-09-25 04:39:57.664 UTC [11365] WARNING: there is no transaction in progress
2026-09-25 04:39:57.675 UTC [11365] WARNING: there is no transaction in progress
2026-09-25 04:39:57.686 UTC [11365] WARNING: there is no transaction in progress
2026-09-25 04:39:57.702 UTC [11365] WARNING: there is no transaction in progress
2026-09-25 04:39:57.712 UTC [11365] WARNING: there is no transaction in progress
2026-09-25 04:39:57.719 UTC [11365] WARNING: there is no transaction in progress
2026-09-25 04:39:57.727 UTC [11365] WARNING: there is no transaction in progress
2026-09-25 04:39:57.736 UTC [11365] WARNING: there is no transaction in progress
2026-09-25 04:39:58,512 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 04:39:58,513 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskilA2AO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskilA2AO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 04:39:58,756 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 04:39:58,756 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskilA2AO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskilA2AO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 04:39:59,022 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 04:39:59,022 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskilA2AO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskilA2AO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 04:39:59,287 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 04:39:59,288 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskilA2AO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskilA2AO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 04:39:59,529 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskilA2AO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskilA2AO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 04:39:59,773 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskilA2AO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskilA2AO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-dc95.c.gcp-upstream.internal';"
2026-09-25 04:40:00,001 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 04:40:00,002 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskilA2AO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskilA2AO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 04:40:00,258 [I] DB_PATCHES have already been applied
I20260925 04:40:00.263681 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskilA2AO/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskilA2AO/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskilA2AO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskilA2AO/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 04:40:00.263754 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260925 04:40:00.267068 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:34775
Sep 25, 2026 4:40:00 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 4:40:00 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 4:40:00 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 4:40:00 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 4:40:00 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 4:40:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 4:40:01 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 04:40:01.283197 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.676s user 0.002s sys 0.008s
Sep 25, 2026 4:40:21 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 4:40:21 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 25, 2026 4:40:21 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 4:40:21 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 4:40:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 4:40:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 4:40:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 4:40:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 4:40:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 4:40:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 4:40:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 4:40:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 4:40:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 4:40:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 4:40:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 4:40:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 4:40:21 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 4:40:22 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 04:40:22.718356 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 04:40:22 dist-test-slave-dc95 krb5kdc[11152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311222, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 04:40:22.820042 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 04:40:22.831033 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 04:40:23.807514 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 04:40:23.807651 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/ranger-kms
I20260925 04:40:23.807807 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:50971
2026-09-25 04:40:23,987 [I] DB FLAVOR :POSTGRES
2026-09-25 04:40:23,988 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 04:40:23,988 [I] Checking connection
2026-09-25 04:40:23,988 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskilA2AO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50971/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 04:40:24,257 [I] connection success
2026-09-25 04:40:24,257 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 04:40:24,257 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 04:40:24,258 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskilA2AO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50971/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 04:40:24,523 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 04:40:24,523 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 04:40:24,523 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 04:40:24,523 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskilA2AO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50971/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 04:40:24,806 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 04:40:25.326229 392 mini_ranger_kms.cc:310] Created kms service
I20260925 04:40:25.531527 392 mini_ranger_kms.cc:326] Created kudu user
I20260925 04:40:25.577731 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 04:40:25 dist-test-slave-dc95 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311222, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 04:40:26.031850 392 mini_ranger_kms.cc:384] Added ranger policy
I20260925 04:40:26.032018 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskilA2AO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskilA2AO/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskilA2AO/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 04:40:26.032042 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260925 04:40:26.035434 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11545
I20260925 04:40:26.035578 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:53929
Sep 25, 2026 4:40:27 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 4:40:27 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 4:40:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 4:40:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 4:40:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 4:40:27 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 04:40:28.181257 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.361s user 0.007s sys 0.157s
I20260925 04:40:28.181401 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 04:40:28.181522 392 mini_ranger_kms.cc:401] 127.0.98.20:53929/kms/v1/keys
Sep 25 04:40:28 dist-test-slave-dc95 krb5kdc[11152](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790311228, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 04:40:29 dist-test-slave-dc95 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311228, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 25 04:40:29 dist-test-slave-dc95 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311228, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 25 04:40:31 dist-test-slave-dc95 krb5kdc[11152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 04:40:31.792578 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35515
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35515
--ranger_config_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:53929/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:40:31.915215 11596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:40:31.915539 11596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:40:31.915596 11596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:40:31.920089 11596 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:40:31.920159 11596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:40:31.920185 11596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:40:31.920207 11596 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:40:31.920238 11596 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:40:31.926168 11596 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:53929/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35515
--ranger_config_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:40:31.927306 11596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:40:31.928416 11596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:40:31.937557 11601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:40:31.937565 11602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:40:31.937750 11596 server_base.cc:1034] running on GCE node
W20260925 04:40:31.937762 11604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:40:31.938664 11596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:40:31.939945 11596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:40:31.941128 11596 hybrid_clock.cc:648] HybridClock initialized: now 1790311231941103 us; error 38 us; skew 500 ppm
Sep 25 04:40:31 dist-test-slave-dc95 krb5kdc[11152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311231, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:40:31.943964 11596 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 04:40:31.944908 11596 webserver.cc:460] Webserver started at http://127.0.98.62:43903/ using document root <none> and password file <none>
I20260925 04:40:31.945436 11596 fs_manager.cc:359] Metadata directory not provided
I20260925 04:40:31.945492 11596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:40:31.945688 11596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 04:40:31 dist-test-slave-dc95 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311231, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 04:40:32.001189 11596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/master-0/data/instance:
uuid: "933b6d73e23c4f259fefbbb50772f11c"
format_stamp: "Formatted at 2026-09-25 04:40:31 on dist-test-slave-dc95"
server_key: "f8b68e366aaab5147b69662ceec30534"
server_key_iv: "263b2f775073abb52d62c80ed9d4893b"
server_key_version: "kuduclusterkey@0"
I20260925 04:40:32.001816 11596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/master-0/wal/instance:
uuid: "933b6d73e23c4f259fefbbb50772f11c"
format_stamp: "Formatted at 2026-09-25 04:40:31 on dist-test-slave-dc95"
server_key: "f8b68e366aaab5147b69662ceec30534"
server_key_iv: "263b2f775073abb52d62c80ed9d4893b"
server_key_version: "kuduclusterkey@0"
I20260925 04:40:32.005919 11596 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 04:40:32.022993 11611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:40:32.024540 11596 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 04:40:32.024698 11596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/master-0/wal
uuid: "933b6d73e23c4f259fefbbb50772f11c"
format_stamp: "Formatted at 2026-09-25 04:40:31 on dist-test-slave-dc95"
server_key: "f8b68e366aaab5147b69662ceec30534"
server_key_iv: "263b2f775073abb52d62c80ed9d4893b"
server_key_version: "kuduclusterkey@0"
I20260925 04:40:32.024824 11596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:40:32.046666 11596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:40:32.049701 11596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:40:32.049897 11596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:40:32.057687 11596 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35515
I20260925 04:40:32.057721 11663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35515 every 8 connection(s)
I20260925 04:40:32.058663 11596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
I20260925 04:40:32.061264 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 11596
I20260925 04:40:32.061385 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/master-0/wal/instance
I20260925 04:40:32.061630 11664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:40:32.061666 392 external_mini_cluster.cc:1428] Setting key d29ca41c40809f3e51434c06c4e92f1e
I20260925 04:40:32.069537 11664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 933b6d73e23c4f259fefbbb50772f11c: Bootstrap starting.
Sep 25 04:40:32 dist-test-slave-dc95 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:40:32.072431 11664 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 933b6d73e23c4f259fefbbb50772f11c: Neither blocks nor log segments found. Creating new log.
I20260925 04:40:32.073311 11664 log.cc:824] T 00000000000000000000000000000000 P 933b6d73e23c4f259fefbbb50772f11c: Log is configured to *not* fsync() on all Append() calls
I20260925 04:40:32.075752 11664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 933b6d73e23c4f259fefbbb50772f11c: No bootstrap required, opened a new log
I20260925 04:40:32.077904 11667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:40:32.063004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45078 (local address 127.0.98.62:35515)
0925 04:40:32.063439 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0925 04:40:32.063457 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 04:40:32.063495 (+ 38us) server_negotiation.cc:407] Connection header received
0925 04:40:32.064453 (+ 958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:40:32.064475 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:40:32.064856 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:40:32.065189 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:40:32.066324 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:40:32.067424 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:40:32.068677 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:40:32.069242 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:40:32.069389 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:40:32.072274 (+ 2885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:40:32.072280 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:40:32.072295 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:40:32.072329 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:40:32.074702 (+ 2373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:40:32.075239 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:40:32.075254 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:40:32.075259 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:40:32.075365 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:40:32.075713 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:40:32.075716 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:40:32.075717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:40:32.076229 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:40:32.076399 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 04:40:32.076761 (+ 362us) server_negotiation.cc:299] Negotiation successful
0925 04:40:32.077112 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":131,"threads_started":1}
I20260925 04:40:32.079142 11664 raft_consensus.cc:348] T 00000000000000000000000000000000 P 933b6d73e23c4f259fefbbb50772f11c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "933b6d73e23c4f259fefbbb50772f11c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35515 } }
I20260925 04:40:32.079433 11664 raft_consensus.cc:374] T 00000000000000000000000000000000 P 933b6d73e23c4f259fefbbb50772f11c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:40:32.079504 11664 raft_consensus.cc:729] T 00000000000000000000000000000000 P 933b6d73e23c4f259fefbbb50772f11c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 933b6d73e23c4f259fefbbb50772f11c, State: Initialized, Role: FOLLOWER
I20260925 04:40:32.080016 11664 consensus_queue.cc:260] T 00000000000000000000000000000000 P 933b6d73e23c4f259fefbbb50772f11c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "933b6d73e23c4f259fefbbb50772f11c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35515 } }
I20260925 04:40:32.080183 11664 raft_consensus.cc:388] T 00000000000000000000000000000000 P 933b6d73e23c4f259fefbbb50772f11c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:40:32.080264 11664 raft_consensus.cc:482] T 00000000000000000000000000000000 P 933b6d73e23c4f259fefbbb50772f11c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:40:32.080365 11664 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 933b6d73e23c4f259fefbbb50772f11c [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:40:32.081332 11664 raft_consensus.cc:504] T 00000000000000000000000000000000 P 933b6d73e23c4f259fefbbb50772f11c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "933b6d73e23c4f259fefbbb50772f11c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35515 } }
I20260925 04:40:32.081665 11664 leader_election.cc:302] T 00000000000000000000000000000000 P 933b6d73e23c4f259fefbbb50772f11c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 933b6d73e23c4f259fefbbb50772f11c; no voters:
I20260925 04:40:32.081998 11664 leader_election.cc:288] T 00000000000000000000000000000000 P 933b6d73e23c4f259fefbbb50772f11c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:40:32.082098 11669 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 933b6d73e23c4f259fefbbb50772f11c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:40:32.082484 11669 raft_consensus.cc:686] T 00000000000000000000000000000000 P 933b6d73e23c4f259fefbbb50772f11c [term 1 LEADER]: Becoming Leader. State: Replica: 933b6d73e23c4f259fefbbb50772f11c, State: Running, Role: LEADER
I20260925 04:40:32.082796 11669 consensus_queue.cc:237] T 00000000000000000000000000000000 P 933b6d73e23c4f259fefbbb50772f11c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "933b6d73e23c4f259fefbbb50772f11c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35515 } }
I20260925 04:40:32.083247 11664 sys_catalog.cc:564] T 00000000000000000000000000000000 P 933b6d73e23c4f259fefbbb50772f11c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:40:32.084523 11671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 933b6d73e23c4f259fefbbb50772f11c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 933b6d73e23c4f259fefbbb50772f11c. Latest consensus state: current_term: 1 leader_uuid: "933b6d73e23c4f259fefbbb50772f11c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "933b6d73e23c4f259fefbbb50772f11c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35515 } } }
I20260925 04:40:32.084575 11670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 933b6d73e23c4f259fefbbb50772f11c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "933b6d73e23c4f259fefbbb50772f11c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "933b6d73e23c4f259fefbbb50772f11c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35515 } } }
I20260925 04:40:32.084633 11671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 933b6d73e23c4f259fefbbb50772f11c [sys.catalog]: This master's current role is: LEADER
I20260925 04:40:32.084693 11670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 933b6d73e23c4f259fefbbb50772f11c [sys.catalog]: This master's current role is: LEADER
I20260925 04:40:32.085091 11681 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:40:32.088395 11681 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:40:32.088799 11664 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 04:40:32.098807 11681 catalog_manager.cc:1347] Generated new cluster ID: 4d77b157974540f9b00ebb61413d3776
I20260925 04:40:32.098929 11681 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:40:32.135525 11681 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:40:32.136583 11681 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:40:32.144094 11681 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 933b6d73e23c4f259fefbbb50772f11c: Generated new TSK 0
I20260925 04:40:32.145285 11681 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 04:40:32.831585 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:53929/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35515
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:40:32.978461 11709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:40:32.978927 11709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:40:32.979012 11709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:40:32.983732 11709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:40:32.983806 11709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:40:32.983891 11709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 04:40:32.989811 11709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:53929/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35515
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:40:32.991001 11709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:40:32.992408 11709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:40:33.004392 11717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:40:33.004613 11716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:40:33.004957 11719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:40:33.005252 11709 server_base.cc:1034] running on GCE node
I20260925 04:40:33.006083 11709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:40:33.007035 11709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:40:33.008226 11709 hybrid_clock.cc:648] HybridClock initialized: now 1790311233008203 us; error 41 us; skew 500 ppm
Sep 25 04:40:33 dist-test-slave-dc95 krb5kdc[11152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311233, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:40:33.011113 11709 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 04:40:33.012415 11709 webserver.cc:460] Webserver started at http://127.0.98.1:45139/ using document root <none> and password file <none>
I20260925 04:40:33.013075 11709 fs_manager.cc:359] Metadata directory not provided
I20260925 04:40:33.013154 11709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:40:33.013366 11709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 04:40:33 dist-test-slave-dc95 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311233, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 04:40:33.030550 11709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/ts-0/data/instance:
uuid: "85c4fff9f22d4c1aa35e67a605399a11"
format_stamp: "Formatted at 2026-09-25 04:40:33 on dist-test-slave-dc95"
server_key: "a8ff637b3c6e996d01ddd293bd66b968"
server_key_iv: "fa5863c7a3bc743fdd83f408ac793063"
server_key_version: "kuduclusterkey@0"
I20260925 04:40:33.031440 11709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/ts-0/wal/instance:
uuid: "85c4fff9f22d4c1aa35e67a605399a11"
format_stamp: "Formatted at 2026-09-25 04:40:33 on dist-test-slave-dc95"
server_key: "a8ff637b3c6e996d01ddd293bd66b968"
server_key_iv: "fa5863c7a3bc743fdd83f408ac793063"
server_key_version: "kuduclusterkey@0"
I20260925 04:40:33.039597 11709 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.001s sys 0.003s
I20260925 04:40:33.076192 11726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:40:33.078102 11709 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.003s sys 0.000s
I20260925 04:40:33.078310 11709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/ts-0/wal
uuid: "85c4fff9f22d4c1aa35e67a605399a11"
format_stamp: "Formatted at 2026-09-25 04:40:33 on dist-test-slave-dc95"
server_key: "a8ff637b3c6e996d01ddd293bd66b968"
server_key_iv: "fa5863c7a3bc743fdd83f408ac793063"
server_key_version: "kuduclusterkey@0"
I20260925 04:40:33.078461 11709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:40:33.100476 11709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:40:33.104456 11709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:40:33.104732 11709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:40:33.105597 11709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:40:33.107136 11709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:40:33.107259 11709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260925 04:40:33.107391 11709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:40:33.107476 11709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:40:33.119138 11709 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44939
I20260925 04:40:33.119205 11841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44939 every 8 connection(s)
I20260925 04:40:33.120584 11709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
I20260925 04:40:33.130405 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 11709
I20260925 04:40:33.130533 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/ts-0/wal/instance
I20260925 04:40:33.130853 392 external_mini_cluster.cc:1428] Setting key 82d549511644b3472bf7f8b9974c9342
Sep 25 04:40:33 dist-test-slave-dc95 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311233, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:40:33.139878 11844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:40:33.122869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41215 (local address 127.0.98.62:35515)
0925 04:40:33.123830 (+ 961us) server_negotiation.cc:206] Beginning negotiation
0925 04:40:33.123840 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 04:40:33.124961 (+ 1121us) server_negotiation.cc:407] Connection header received
0925 04:40:33.126212 (+ 1251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:40:33.126216 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:40:33.126281 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:40:33.126394 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:40:33.128003 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:40:33.129024 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:40:33.131041 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:40:33.131467 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:40:33.131596 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:40:33.135026 (+ 3430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:40:33.135031 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:40:33.135034 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:40:33.135066 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:40:33.137038 (+ 1972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:40:33.137852 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:40:33.137856 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:40:33.137858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:40:33.137963 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:40:33.138412 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:40:33.138415 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:40:33.138417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:40:33.138698 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:40:33.138790 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 04:40:33.139495 (+ 705us) server_negotiation.cc:299] Negotiation successful
0925 04:40:33.139680 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":868,"thread_start_us":107,"threads_started":1}
I20260925 04:40:33.141075 11845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:40:33.123425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35515 (local address 127.0.98.1:41215)
0925 04:40:33.124730 (+ 1305us) negotiation.cc:107] Waiting for socket to connect
0925 04:40:33.124790 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 04:40:33.125934 (+ 1144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:40:33.126630 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:40:33.126641 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:40:33.127031 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:40:33.127811 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:40:33.127836 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:40:33.129205 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:40:33.129209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:40:33.130860 (+ 1651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:40:33.130875 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:40:33.131937 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:40:33.131941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:40:33.132190 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:40:33.132975 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:40:33.132999 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:40:33.134765 (+ 1766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:40:33.137205 (+ 2440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:40:33.137213 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:40:33.137236 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:40:33.137682 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:40:33.138066 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:40:33.138069 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:40:33.138072 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:40:33.138298 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:40:33.138814 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:40:33.138822 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:40:33.139197 (+ 375us) client_negotiation.cc:769] Sending connection context
0925 04:40:33.139409 (+ 212us) client_negotiation.cc:240] Negotiation successful
0925 04:40:33.139833 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1076,"thread_start_us":214,"threads_started":1}
I20260925 04:40:33.142714 11842 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35515
I20260925 04:40:33.143038 11842 heartbeater.cc:464] Registering TS with master...
I20260925 04:40:33.143746 11842 heartbeater.cc:511] Master 127.0.98.62:35515 requested a full tablet report, sending...
I20260925 04:40:33.145788 11628 ts_manager.cc:194] Registered new tserver with Master: 85c4fff9f22d4c1aa35e67a605399a11 (127.0.98.1:44939)
I20260925 04:40:33.147428 11628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41215
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Sep 25 04:40:33 dist-test-slave-dc95 krb5kdc[11152](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790311233, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 04:40:33.175462 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:53929/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35515
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:40:33.310535 11848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:40:33.310839 11848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:40:33.310901 11848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:40:33.315830 11848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:40:33.315918 11848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:40:33.316006 11848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 04:40:33.322619 11848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:53929/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35515
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:40:33.323927 11848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:40:33.325066 11848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 04:40:33.337783 11848 server_base.cc:1034] running on GCE node
W20260925 04:40:33.338104 11855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:40:33.338690 11854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:40:33.340224 11857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:40:33.340867 11848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:40:33.341936 11848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:40:33.343194 11848 hybrid_clock.cc:648] HybridClock initialized: now 1790311233343158 us; error 50 us; skew 500 ppm
Sep 25 04:40:33 dist-test-slave-dc95 krb5kdc[11152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311233, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:40:33.347661 11848 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 04:40:33.349054 11848 webserver.cc:460] Webserver started at http://127.0.98.2:35085/ using document root <none> and password file <none>
I20260925 04:40:33.349681 11848 fs_manager.cc:359] Metadata directory not provided
I20260925 04:40:33.349752 11848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:40:33.349931 11848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 04:40:33 dist-test-slave-dc95 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311233, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 04:40:33.368249 11848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/ts-1/data/instance:
uuid: "b0f6866121ef4939b5c5139bc8165b63"
format_stamp: "Formatted at 2026-09-25 04:40:33 on dist-test-slave-dc95"
server_key: "e91f1669143c6a6561f49b27f61fe6b3"
server_key_iv: "6bf50c11020afc1a9689ecac2293395d"
server_key_version: "kuduclusterkey@0"
I20260925 04:40:33.370345 11848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b0f6866121ef4939b5c5139bc8165b63"
format_stamp: "Formatted at 2026-09-25 04:40:33 on dist-test-slave-dc95"
server_key: "e91f1669143c6a6561f49b27f61fe6b3"
server_key_iv: "6bf50c11020afc1a9689ecac2293395d"
server_key_version: "kuduclusterkey@0"
I20260925 04:40:33.376482 11848 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260925 04:40:33.402307 11864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:40:33.404290 11848 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.004s sys 0.000s
I20260925 04:40:33.404451 11848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/ts-1/wal
uuid: "b0f6866121ef4939b5c5139bc8165b63"
format_stamp: "Formatted at 2026-09-25 04:40:33 on dist-test-slave-dc95"
server_key: "e91f1669143c6a6561f49b27f61fe6b3"
server_key_iv: "6bf50c11020afc1a9689ecac2293395d"
server_key_version: "kuduclusterkey@0"
I20260925 04:40:33.404597 11848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:40:33.426568 11848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:40:33.430696 11848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:40:33.430974 11848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:40:33.431807 11848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:40:33.433190 11848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:40:33.433306 11848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260925 04:40:33.433404 11848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:40:33.433460 11848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:40:33.445940 11848 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37607
I20260925 04:40:33.448372 11848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
I20260925 04:40:33.451027 11977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37607 every 8 connection(s)
I20260925 04:40:33.455103 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 11848
I20260925 04:40:33.455237 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/ts-1/wal/instance
I20260925 04:40:33.455538 392 external_mini_cluster.cc:1428] Setting key c3353c433e16404f4bdeb10ddc35cc99
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Sep 25 04:40:33 dist-test-slave-dc95 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311233, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:40:33.489882 11844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:40:33.455775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60101 (local address 127.0.98.62:35515)
0925 04:40:33.455948 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 04:40:33.455953 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:40:33.464685 (+ 8732us) server_negotiation.cc:407] Connection header received
0925 04:40:33.472071 (+ 7386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:40:33.472100 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:40:33.472229 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:40:33.472348 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:40:33.477101 (+ 4753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:40:33.477984 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:40:33.480183 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:40:33.480549 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:40:33.480663 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:40:33.484472 (+ 3809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:40:33.484478 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:40:33.484481 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:40:33.484521 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:40:33.486299 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:40:33.487168 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:40:33.487173 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:40:33.487175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:40:33.487280 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:40:33.487793 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:40:33.487797 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:40:33.487799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:40:33.488076 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:40:33.488181 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 04:40:33.489520 (+ 1339us) server_negotiation.cc:299] Negotiation successful
0925 04:40:33.489697 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 04:40:33.491890 11980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:40:33.453616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35515 (local address 127.0.98.2:60101)
0925 04:40:33.464307 (+ 10691us) negotiation.cc:107] Waiting for socket to connect
0925 04:40:33.464459 (+ 152us) client_negotiation.cc:174] Beginning negotiation
0925 04:40:33.466484 (+ 2025us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:40:33.473300 (+ 6816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:40:33.473314 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:40:33.473771 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:40:33.474657 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:40:33.474692 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:40:33.478230 (+ 3538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:40:33.478234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:40:33.479971 (+ 1737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:40:33.479993 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:40:33.480932 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:40:33.480936 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:40:33.481185 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:40:33.481951 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:40:33.481975 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:40:33.484185 (+ 2210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:40:33.486549 (+ 2364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:40:33.486558 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:40:33.486580 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:40:33.487003 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:40:33.487395 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:40:33.487399 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:40:33.487401 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:40:33.487666 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:40:33.488263 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:40:33.488272 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:40:33.488693 (+ 421us) client_negotiation.cc:769] Sending connection context
0925 04:40:33.490170 (+ 1477us) client_negotiation.cc:240] Negotiation successful
0925 04:40:33.490609 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10206,"thread_start_us":174,"threads_started":1}
I20260925 04:40:33.493628 11978 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35515
I20260925 04:40:33.493943 11978 heartbeater.cc:464] Registering TS with master...
I20260925 04:40:33.494544 11978 heartbeater.cc:511] Master 127.0.98.62:35515 requested a full tablet report, sending...
I20260925 04:40:33.495909 11628 ts_manager.cc:194] Registered new tserver with Master: b0f6866121ef4939b5c5139bc8165b63 (127.0.98.2:37607)
I20260925 04:40:33.496767 11628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60101
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 04:40:33.505573 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:53929/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35515
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:40:33.675004 11983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:40:33.675482 11983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:40:33.675639 11983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:40:33.681973 11983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:40:33.682232 11983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:40:33.682389 11983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 04:40:33.691347 11983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:53929/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35515
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:40:33.693090 11983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:40:33.694571 11983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:40:33.710099 11989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:40:33.710493 11983 server_base.cc:1034] running on GCE node
W20260925 04:40:33.711129 11988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:40:33.712291 11991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:40:33.712949 11983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:40:33.714259 11983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:40:33.717592 11983 hybrid_clock.cc:648] HybridClock initialized: now 1790311233717492 us; error 102 us; skew 500 ppm
Sep 25 04:40:33 dist-test-slave-dc95 krb5kdc[11152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311233, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:40:33.722589 11983 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 04:40:33.723996 11983 webserver.cc:460] Webserver started at http://127.0.98.3:36543/ using document root <none> and password file <none>
I20260925 04:40:33.726393 11983 fs_manager.cc:359] Metadata directory not provided
I20260925 04:40:33.726523 11983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:40:33.726837 11983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 04:40:33 dist-test-slave-dc95 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311233, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 04:40:33.753367 11983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/ts-2/data/instance:
uuid: "501de8d019da4b8c91035dafd82c6464"
format_stamp: "Formatted at 2026-09-25 04:40:33 on dist-test-slave-dc95"
server_key: "9ba4417fdac7910f79b6d0d84a74c11d"
server_key_iv: "641a7323cad63f6dbfb1d2e8b04488bf"
server_key_version: "kuduclusterkey@0"
I20260925 04:40:33.754647 11983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/ts-2/wal/instance:
uuid: "501de8d019da4b8c91035dafd82c6464"
format_stamp: "Formatted at 2026-09-25 04:40:33 on dist-test-slave-dc95"
server_key: "9ba4417fdac7910f79b6d0d84a74c11d"
server_key_iv: "641a7323cad63f6dbfb1d2e8b04488bf"
server_key_version: "kuduclusterkey@0"
I20260925 04:40:33.760540 11983 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 04:40:33.792621 11998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:40:33.794373 11983 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260925 04:40:33.794529 11983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/ts-2/wal
uuid: "501de8d019da4b8c91035dafd82c6464"
format_stamp: "Formatted at 2026-09-25 04:40:33 on dist-test-slave-dc95"
server_key: "9ba4417fdac7910f79b6d0d84a74c11d"
server_key_iv: "641a7323cad63f6dbfb1d2e8b04488bf"
server_key_version: "kuduclusterkey@0"
I20260925 04:40:33.794679 11983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 25 04:40:33 dist-test-slave-dc95 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311233, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 04:40:33.826828 11983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:40:33.830912 11983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:40:33.831115 11983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:40:33.831835 11983 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:40:33.832906 11983 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:40:33.832983 11983 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:40:33.833045 11983 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:40:33.833072 11983 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:40:33.847792 11983 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46183
I20260925 04:40:33.849074 11983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
I20260925 04:40:33.849920 12111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46183 every 8 connection(s)
I20260925 04:40:33.850528 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 11983
I20260925 04:40:33.850652 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/ts-2/wal/instance
I20260925 04:40:33.850934 392 external_mini_cluster.cc:1428] Setting key b18e6b55f0edbb25539cfaf2605eeb37
Sep 25 04:40:33 dist-test-slave-dc95 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311233, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:40:33.870226 11844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:40:33.855625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58337 (local address 127.0.98.62:35515)
0925 04:40:33.855765 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 04:40:33.855769 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:40:33.856200 (+ 431us) server_negotiation.cc:407] Connection header received
0925 04:40:33.857637 (+ 1437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:40:33.857641 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:40:33.857705 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:40:33.857755 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:40:33.859392 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:40:33.860239 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:40:33.861876 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:40:33.862214 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:40:33.862285 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:40:33.865718 (+ 3433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:40:33.865723 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:40:33.865725 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:40:33.865755 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:40:33.867361 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:40:33.868344 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:40:33.868348 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:40:33.868350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:40:33.868446 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:40:33.868852 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:40:33.868855 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:40:33.868857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:40:33.869078 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:40:33.869162 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 04:40:33.869928 (+ 766us) server_negotiation.cc:299] Negotiation successful
0925 04:40:33.870054 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 04:40:33.872092 12114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:40:33.854472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35515 (local address 127.0.98.3:58337)
0925 04:40:33.856032 (+ 1560us) negotiation.cc:107] Waiting for socket to connect
0925 04:40:33.856094 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0925 04:40:33.857393 (+ 1299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:40:33.858194 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:40:33.858202 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:40:33.858551 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:40:33.859221 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:40:33.859242 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:40:33.860400 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:40:33.860403 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:40:33.861736 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:40:33.861748 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:40:33.862493 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:40:33.862497 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:40:33.862705 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:40:33.863739 (+ 1034us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:40:33.863770 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:40:33.865496 (+ 1726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:40:33.867513 (+ 2017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:40:33.867519 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:40:33.867540 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:40:33.867936 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:40:33.868552 (+ 616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:40:33.868555 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:40:33.868558 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:40:33.868751 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:40:33.869294 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:40:33.869301 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:40:33.869654 (+ 353us) client_negotiation.cc:769] Sending connection context
0925 04:40:33.870670 (+ 1016us) client_negotiation.cc:240] Negotiation successful
0925 04:40:33.871029 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1316,"thread_start_us":164,"threads_started":1}
I20260925 04:40:33.873690 12112 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35515
I20260925 04:40:33.873930 12112 heartbeater.cc:464] Registering TS with master...
I20260925 04:40:33.874491 12112 heartbeater.cc:511] Master 127.0.98.62:35515 requested a full tablet report, sending...
I20260925 04:40:33.875723 11628 ts_manager.cc:194] Registered new tserver with Master: 501de8d019da4b8c91035dafd82c6464 (127.0.98.3:46183)
I20260925 04:40:33.876506 11628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58337
I20260925 04:40:33.881174 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:40:33.901918 11844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:40:33.884180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45094 (local address 127.0.98.62:35515)
0925 04:40:33.884325 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 04:40:33.884329 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:40:33.892101 (+ 7772us) server_negotiation.cc:407] Connection header received
0925 04:40:33.892340 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:40:33.892344 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:40:33.892408 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:40:33.892463 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:40:33.893483 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:40:33.894275 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:40:33.895474 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:40:33.895867 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:40:33.895951 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:40:33.897813 (+ 1862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:40:33.897817 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:40:33.897820 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:40:33.897849 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:40:33.899872 (+ 2023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:40:33.900449 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:40:33.900453 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:40:33.900455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:40:33.900543 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:40:33.900834 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:40:33.900837 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:40:33.900838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:40:33.901058 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:40:33.901191 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 04:40:33.901618 (+ 427us) server_negotiation.cc:299] Negotiation successful
0925 04:40:33.901757 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 04:40:33.906877 11628 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45094:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:40:33.913252 11628 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:40:33.945345 12124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:40:33.925022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44939 (local address 127.0.0.1:60348)
0925 04:40:33.929646 (+ 4624us) negotiation.cc:107] Waiting for socket to connect
0925 04:40:33.929708 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0925 04:40:33.930103 (+ 395us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:40:33.931656 (+ 1553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:40:33.931664 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:40:33.931691 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:40:33.931848 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:40:33.931855 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:40:33.933607 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:40:33.933611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:40:33.935096 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:40:33.935108 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:40:33.944699 (+ 9591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:40:33.944702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:40:33.944858 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:40:33.944902 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 04:40:33.945020 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 04:40:33.945169 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4423,"thread_start_us":129,"threads_started":1}
I20260925 04:40:33.949947 12128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:40:33.927995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46183 (local address 127.0.0.1:42706)
0925 04:40:33.932062 (+ 4067us) negotiation.cc:107] Waiting for socket to connect
0925 04:40:33.932074 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:40:33.932140 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:40:33.936157 (+ 4017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:40:33.936161 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:40:33.936181 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:40:33.936500 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:40:33.936508 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:40:33.939049 (+ 2541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:40:33.939052 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:40:33.940311 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:40:33.940324 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:40:33.949422 (+ 9098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:40:33.949427 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:40:33.949535 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:40:33.949563 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 04:40:33.949640 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 04:40:33.949733 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4009,"thread_start_us":91,"threads_started":1}
I20260925 04:40:33.950845 12125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:40:33.925952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60348 (local address 127.0.98.1:44939)
0925 04:40:33.929939 (+ 3987us) server_negotiation.cc:206] Beginning negotiation
0925 04:40:33.929948 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:40:33.930251 (+ 303us) server_negotiation.cc:407] Connection header received
0925 04:40:33.930336 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:40:33.930343 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:40:33.930577 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:40:33.930711 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:40:33.932258 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:40:33.933467 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:40:33.943337 (+ 9870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:40:33.944339 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:40:33.944459 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:40:33.944629 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 04:40:33.950551 (+ 5922us) server_negotiation.cc:299] Negotiation successful
0925 04:40:33.950695 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3874,"thread_start_us":92,"threads_started":1}
I20260925 04:40:33.954777 11776 tablet_service.cc:1467] Processing CreateTablet for tablet 39f8ec4d909b49f0b51dfd86ebaf589f (DEFAULT_TABLE table=test-table [id=e3b16ad74b86458bbf08200852414aad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:40:33.957676 11776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39f8ec4d909b49f0b51dfd86ebaf589f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:40:33.961404 12129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:40:33.928410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42706 (local address 127.0.98.3:46183)
0925 04:40:33.935439 (+ 7029us) server_negotiation.cc:206] Beginning negotiation
0925 04:40:33.935450 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 04:40:33.935475 (+ 25us) server_negotiation.cc:407] Connection header received
0925 04:40:33.935558 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:40:33.935564 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:40:33.935844 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:40:33.936008 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:40:33.937686 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:40:33.938902 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:40:33.948046 (+ 9144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:40:33.948993 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:40:33.949140 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:40:33.949334 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0925 04:40:33.961045 (+ 11711us) server_negotiation.cc:299] Negotiation successful
0925 04:40:33.961229 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6900,"thread_start_us":111,"threads_started":1}
I20260925 04:40:33.962529 12126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:40:33.926770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37607 (local address 127.0.0.1:33730)
0925 04:40:33.930960 (+ 4190us) negotiation.cc:107] Waiting for socket to connect
0925 04:40:33.930974 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:40:33.931054 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:40:33.937423 (+ 6369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:40:33.937426 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:40:33.937446 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:40:33.937554 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:40:33.937559 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:40:33.941780 (+ 4221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:40:33.941783 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:40:33.943161 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:40:33.943174 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:40:33.961954 (+ 18780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:40:33.961960 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:40:33.962110 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:40:33.962146 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 04:40:33.962281 (+ 135us) client_negotiation.cc:240] Negotiation successful
0925 04:40:33.962377 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4125,"thread_start_us":94,"threads_started":1}
I20260925 04:40:33.963150 12127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:40:33.927201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33730 (local address 127.0.98.2:37607)
0925 04:40:33.936781 (+ 9580us) server_negotiation.cc:206] Beginning negotiation
0925 04:40:33.936798 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 04:40:33.936823 (+ 25us) server_negotiation.cc:407] Connection header received
0925 04:40:33.936898 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:40:33.936903 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:40:33.937120 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:40:33.937281 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:40:33.940485 (+ 3204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:40:33.941635 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:40:33.946018 (+ 4383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:40:33.960100 (+ 14082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:40:33.960373 (+ 273us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:40:33.960763 (+ 390us) server_negotiation.cc:1035] Waiting for connection context
0925 04:40:33.962818 (+ 2055us) server_negotiation.cc:299] Negotiation successful
0925 04:40:33.962966 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9460,"thread_start_us":121,"threads_started":1}
I20260925 04:40:33.965090 12046 tablet_service.cc:1467] Processing CreateTablet for tablet 39f8ec4d909b49f0b51dfd86ebaf589f (DEFAULT_TABLE table=test-table [id=e3b16ad74b86458bbf08200852414aad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:40:33.966368 12046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39f8ec4d909b49f0b51dfd86ebaf589f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:40:33.967144 11912 tablet_service.cc:1467] Processing CreateTablet for tablet 39f8ec4d909b49f0b51dfd86ebaf589f (DEFAULT_TABLE table=test-table [id=e3b16ad74b86458bbf08200852414aad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:40:33.968854 11912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39f8ec4d909b49f0b51dfd86ebaf589f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:40:33.975966 12130 tablet_bootstrap.cc:492] T 39f8ec4d909b49f0b51dfd86ebaf589f P 501de8d019da4b8c91035dafd82c6464: Bootstrap starting.
I20260925 04:40:33.979010 12130 tablet_bootstrap.cc:654] T 39f8ec4d909b49f0b51dfd86ebaf589f P 501de8d019da4b8c91035dafd82c6464: Neither blocks nor log segments found. Creating new log.
I20260925 04:40:33.980010 12130 log.cc:824] T 39f8ec4d909b49f0b51dfd86ebaf589f P 501de8d019da4b8c91035dafd82c6464: Log is configured to *not* fsync() on all Append() calls
I20260925 04:40:33.982472 12130 tablet_bootstrap.cc:492] T 39f8ec4d909b49f0b51dfd86ebaf589f P 501de8d019da4b8c91035dafd82c6464: No bootstrap required, opened a new log
I20260925 04:40:33.982695 12130 ts_tablet_manager.cc:1397] T 39f8ec4d909b49f0b51dfd86ebaf589f P 501de8d019da4b8c91035dafd82c6464: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260925 04:40:33.987293 12130 raft_consensus.cc:348] T 39f8ec4d909b49f0b51dfd86ebaf589f P 501de8d019da4b8c91035dafd82c6464 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85c4fff9f22d4c1aa35e67a605399a11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44939 } } peers { permanent_uuid: "b0f6866121ef4939b5c5139bc8165b63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37607 } } peers { permanent_uuid: "501de8d019da4b8c91035dafd82c6464" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46183 } }
I20260925 04:40:33.987659 12130 raft_consensus.cc:374] T 39f8ec4d909b49f0b51dfd86ebaf589f P 501de8d019da4b8c91035dafd82c6464 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:40:33.987712 12130 raft_consensus.cc:729] T 39f8ec4d909b49f0b51dfd86ebaf589f P 501de8d019da4b8c91035dafd82c6464 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 501de8d019da4b8c91035dafd82c6464, State: Initialized, Role: FOLLOWER
I20260925 04:40:33.988234 12130 consensus_queue.cc:260] T 39f8ec4d909b49f0b51dfd86ebaf589f P 501de8d019da4b8c91035dafd82c6464 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85c4fff9f22d4c1aa35e67a605399a11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44939 } } peers { permanent_uuid: "b0f6866121ef4939b5c5139bc8165b63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37607 } } peers { permanent_uuid: "501de8d019da4b8c91035dafd82c6464" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46183 } }
I20260925 04:40:33.988319 12132 tablet_bootstrap.cc:492] T 39f8ec4d909b49f0b51dfd86ebaf589f P 85c4fff9f22d4c1aa35e67a605399a11: Bootstrap starting.
I20260925 04:40:33.989295 12130 ts_tablet_manager.cc:1428] T 39f8ec4d909b49f0b51dfd86ebaf589f P 501de8d019da4b8c91035dafd82c6464: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260925 04:40:33.989706 12112 heartbeater.cc:503] Master 127.0.98.62:35515 was elected leader, sending a full tablet report...
I20260925 04:40:33.991108 12132 tablet_bootstrap.cc:654] T 39f8ec4d909b49f0b51dfd86ebaf589f P 85c4fff9f22d4c1aa35e67a605399a11: Neither blocks nor log segments found. Creating new log.
I20260925 04:40:33.992128 12132 log.cc:824] T 39f8ec4d909b49f0b51dfd86ebaf589f P 85c4fff9f22d4c1aa35e67a605399a11: Log is configured to *not* fsync() on all Append() calls
I20260925 04:40:33.993350 12133 tablet_bootstrap.cc:492] T 39f8ec4d909b49f0b51dfd86ebaf589f P b0f6866121ef4939b5c5139bc8165b63: Bootstrap starting.
I20260925 04:40:33.996295 12133 tablet_bootstrap.cc:654] T 39f8ec4d909b49f0b51dfd86ebaf589f P b0f6866121ef4939b5c5139bc8165b63: Neither blocks nor log segments found. Creating new log.
I20260925 04:40:33.997289 12133 log.cc:824] T 39f8ec4d909b49f0b51dfd86ebaf589f P b0f6866121ef4939b5c5139bc8165b63: Log is configured to *not* fsync() on all Append() calls
I20260925 04:40:33.997903 12132 tablet_bootstrap.cc:492] T 39f8ec4d909b49f0b51dfd86ebaf589f P 85c4fff9f22d4c1aa35e67a605399a11: No bootstrap required, opened a new log
I20260925 04:40:33.998149 12132 ts_tablet_manager.cc:1397] T 39f8ec4d909b49f0b51dfd86ebaf589f P 85c4fff9f22d4c1aa35e67a605399a11: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260925 04:40:33.999790 12133 tablet_bootstrap.cc:492] T 39f8ec4d909b49f0b51dfd86ebaf589f P b0f6866121ef4939b5c5139bc8165b63: No bootstrap required, opened a new log
I20260925 04:40:33.999998 12133 ts_tablet_manager.cc:1397] T 39f8ec4d909b49f0b51dfd86ebaf589f P b0f6866121ef4939b5c5139bc8165b63: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260925 04:40:34.003932 12132 raft_consensus.cc:348] T 39f8ec4d909b49f0b51dfd86ebaf589f P 85c4fff9f22d4c1aa35e67a605399a11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85c4fff9f22d4c1aa35e67a605399a11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44939 } } peers { permanent_uuid: "b0f6866121ef4939b5c5139bc8165b63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37607 } } peers { permanent_uuid: "501de8d019da4b8c91035dafd82c6464" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46183 } }
I20260925 04:40:34.004304 12132 raft_consensus.cc:374] T 39f8ec4d909b49f0b51dfd86ebaf589f P 85c4fff9f22d4c1aa35e67a605399a11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:40:34.004374 12132 raft_consensus.cc:729] T 39f8ec4d909b49f0b51dfd86ebaf589f P 85c4fff9f22d4c1aa35e67a605399a11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85c4fff9f22d4c1aa35e67a605399a11, State: Initialized, Role: FOLLOWER
I20260925 04:40:34.004957 12132 consensus_queue.cc:260] T 39f8ec4d909b49f0b51dfd86ebaf589f P 85c4fff9f22d4c1aa35e67a605399a11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85c4fff9f22d4c1aa35e67a605399a11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44939 } } peers { permanent_uuid: "b0f6866121ef4939b5c5139bc8165b63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37607 } } peers { permanent_uuid: "501de8d019da4b8c91035dafd82c6464" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46183 } }
I20260925 04:40:34.005604 12133 raft_consensus.cc:348] T 39f8ec4d909b49f0b51dfd86ebaf589f P b0f6866121ef4939b5c5139bc8165b63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85c4fff9f22d4c1aa35e67a605399a11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44939 } } peers { permanent_uuid: "b0f6866121ef4939b5c5139bc8165b63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37607 } } peers { permanent_uuid: "501de8d019da4b8c91035dafd82c6464" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46183 } }
I20260925 04:40:34.006064 12133 raft_consensus.cc:374] T 39f8ec4d909b49f0b51dfd86ebaf589f P b0f6866121ef4939b5c5139bc8165b63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:40:34.006096 12132 ts_tablet_manager.cc:1428] T 39f8ec4d909b49f0b51dfd86ebaf589f P 85c4fff9f22d4c1aa35e67a605399a11: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260925 04:40:34.006194 12133 raft_consensus.cc:729] T 39f8ec4d909b49f0b51dfd86ebaf589f P b0f6866121ef4939b5c5139bc8165b63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0f6866121ef4939b5c5139bc8165b63, State: Initialized, Role: FOLLOWER
I20260925 04:40:34.006474 11842 heartbeater.cc:503] Master 127.0.98.62:35515 was elected leader, sending a full tablet report...
I20260925 04:40:34.006829 12133 consensus_queue.cc:260] T 39f8ec4d909b49f0b51dfd86ebaf589f P b0f6866121ef4939b5c5139bc8165b63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85c4fff9f22d4c1aa35e67a605399a11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44939 } } peers { permanent_uuid: "b0f6866121ef4939b5c5139bc8165b63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37607 } } peers { permanent_uuid: "501de8d019da4b8c91035dafd82c6464" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46183 } }
I20260925 04:40:34.008023 12133 ts_tablet_manager.cc:1428] T 39f8ec4d909b49f0b51dfd86ebaf589f P b0f6866121ef4939b5c5139bc8165b63: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260925 04:40:34.008451 11978 heartbeater.cc:503] Master 127.0.98.62:35515 was elected leader, sending a full tablet report...
I20260925 04:40:34.070138 12137 raft_consensus.cc:482] T 39f8ec4d909b49f0b51dfd86ebaf589f P 85c4fff9f22d4c1aa35e67a605399a11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:40:34.070374 12137 raft_consensus.cc:504] T 39f8ec4d909b49f0b51dfd86ebaf589f P 85c4fff9f22d4c1aa35e67a605399a11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85c4fff9f22d4c1aa35e67a605399a11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44939 } } peers { permanent_uuid: "b0f6866121ef4939b5c5139bc8165b63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37607 } } peers { permanent_uuid: "501de8d019da4b8c91035dafd82c6464" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46183 } }
I20260925 04:40:34.071533 12137 leader_election.cc:288] T 39f8ec4d909b49f0b51dfd86ebaf589f P 85c4fff9f22d4c1aa35e67a605399a11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0f6866121ef4939b5c5139bc8165b63 (127.0.98.2:37607), 501de8d019da4b8c91035dafd82c6464 (127.0.98.3:46183)
I20260925 04:40:34.077181 12127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:40:34.071910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51967 (local address 127.0.98.2:37607)
0925 04:40:34.072159 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0925 04:40:34.072164 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:40:34.072909 (+ 745us) server_negotiation.cc:407] Connection header received
0925 04:40:34.073140 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:40:34.073144 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:40:34.073191 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:40:34.073235 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:40:34.073634 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:40:34.074310 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:40:34.075732 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:40:34.076362 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:40:34.076439 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:40:34.076510 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 04:40:34.076965 (+ 455us) server_negotiation.cc:299] Negotiation successful
0925 04:40:34.077046 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160}
I20260925 04:40:34.077175 12129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:40:34.072277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57045 (local address 127.0.98.3:46183)
0925 04:40:34.072392 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0925 04:40:34.072398 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:40:34.072444 (+ 46us) server_negotiation.cc:407] Connection header received
0925 04:40:34.072664 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:40:34.072669 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:40:34.072722 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:40:34.072787 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:40:34.073232 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:40:34.073998 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:40:34.075733 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:40:34.076352 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:40:34.076439 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:40:34.076510 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 04:40:34.076936 (+ 426us) server_negotiation.cc:299] Negotiation successful
0925 04:40:34.077026 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 04:40:34.077668 12140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:40:34.072471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37607 (local address 127.0.98.1:51967)
0925 04:40:34.072832 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0925 04:40:34.072846 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:40:34.073020 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:40:34.073378 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:40:34.073381 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:40:34.073398 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:40:34.073515 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:40:34.073521 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:40:34.074434 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:40:34.074436 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:40:34.075549 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:40:34.075558 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:40:34.076615 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:40:34.076617 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:40:34.076841 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:40:34.076859 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 04:40:34.077416 (+ 557us) client_negotiation.cc:240] Negotiation successful
0925 04:40:34.077506 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":98,"spinlock_wait_cycles":20864,"thread_start_us":104,"threads_started":1}
I20260925 04:40:34.078238 12139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:40:34.071905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46183 (local address 127.0.98.1:57045)
0925 04:40:34.072355 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0925 04:40:34.072370 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:40:34.072528 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:40:34.072938 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:40:34.072941 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:40:34.072958 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:40:34.073115 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:40:34.073122 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:40:34.074157 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:40:34.074160 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:40:34.075546 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:40:34.075558 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:40:34.076610 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:40:34.076614 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:40:34.076784 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:40:34.076808 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 04:40:34.078003 (+ 1195us) client_negotiation.cc:240] Negotiation successful
0925 04:40:34.078087 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"mutex_wait_us":49,"spinlock_wait_cycles":21760,"thread_start_us":101,"threads_started":1}
I20260925 04:40:34.078369 11921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39f8ec4d909b49f0b51dfd86ebaf589f" candidate_uuid: "85c4fff9f22d4c1aa35e67a605399a11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0f6866121ef4939b5c5139bc8165b63" is_pre_election: true
I20260925 04:40:34.078722 11921 raft_consensus.cc:2454] T 39f8ec4d909b49f0b51dfd86ebaf589f P b0f6866121ef4939b5c5139bc8165b63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85c4fff9f22d4c1aa35e67a605399a11 in term 0.
I20260925 04:40:34.079033 12050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39f8ec4d909b49f0b51dfd86ebaf589f" candidate_uuid: "85c4fff9f22d4c1aa35e67a605399a11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "501de8d019da4b8c91035dafd82c6464" is_pre_election: true
I20260925 04:40:34.079245 11731 leader_election.cc:302] T 39f8ec4d909b49f0b51dfd86ebaf589f P 85c4fff9f22d4c1aa35e67a605399a11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85c4fff9f22d4c1aa35e67a605399a11, b0f6866121ef4939b5c5139bc8165b63; no voters:
I20260925 04:40:34.079355 12050 raft_consensus.cc:2454] T 39f8ec4d909b49f0b51dfd86ebaf589f P 501de8d019da4b8c91035dafd82c6464 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85c4fff9f22d4c1aa35e67a605399a11 in term 0.
I20260925 04:40:34.079629 12137 raft_consensus.cc:2781] T 39f8ec4d909b49f0b51dfd86ebaf589f P 85c4fff9f22d4c1aa35e67a605399a11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:40:34.079733 12137 raft_consensus.cc:482] T 39f8ec4d909b49f0b51dfd86ebaf589f P 85c4fff9f22d4c1aa35e67a605399a11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:40:34.079804 12137 raft_consensus.cc:3037] T 39f8ec4d909b49f0b51dfd86ebaf589f P 85c4fff9f22d4c1aa35e67a605399a11 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:40:34.081255 12137 raft_consensus.cc:504] T 39f8ec4d909b49f0b51dfd86ebaf589f P 85c4fff9f22d4c1aa35e67a605399a11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85c4fff9f22d4c1aa35e67a605399a11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44939 } } peers { permanent_uuid: "b0f6866121ef4939b5c5139bc8165b63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37607 } } peers { permanent_uuid: "501de8d019da4b8c91035dafd82c6464" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46183 } }
I20260925 04:40:34.081710 12137 leader_election.cc:288] T 39f8ec4d909b49f0b51dfd86ebaf589f P 85c4fff9f22d4c1aa35e67a605399a11 [CANDIDATE]: Term 1 election: Requested vote from peers b0f6866121ef4939b5c5139bc8165b63 (127.0.98.2:37607), 501de8d019da4b8c91035dafd82c6464 (127.0.98.3:46183)
I20260925 04:40:34.082257 11921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39f8ec4d909b49f0b51dfd86ebaf589f" candidate_uuid: "85c4fff9f22d4c1aa35e67a605399a11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0f6866121ef4939b5c5139bc8165b63"
I20260925 04:40:34.082309 12050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39f8ec4d909b49f0b51dfd86ebaf589f" candidate_uuid: "85c4fff9f22d4c1aa35e67a605399a11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "501de8d019da4b8c91035dafd82c6464"
I20260925 04:40:34.082402 11921 raft_consensus.cc:3037] T 39f8ec4d909b49f0b51dfd86ebaf589f P b0f6866121ef4939b5c5139bc8165b63 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:40:34.082442 12050 raft_consensus.cc:3037] T 39f8ec4d909b49f0b51dfd86ebaf589f P 501de8d019da4b8c91035dafd82c6464 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:40:34.083892 11921 raft_consensus.cc:2454] T 39f8ec4d909b49f0b51dfd86ebaf589f P b0f6866121ef4939b5c5139bc8165b63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85c4fff9f22d4c1aa35e67a605399a11 in term 1.
I20260925 04:40:34.084079 12050 raft_consensus.cc:2454] T 39f8ec4d909b49f0b51dfd86ebaf589f P 501de8d019da4b8c91035dafd82c6464 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85c4fff9f22d4c1aa35e67a605399a11 in term 1.
I20260925 04:40:34.084543 11729 leader_election.cc:302] T 39f8ec4d909b49f0b51dfd86ebaf589f P 85c4fff9f22d4c1aa35e67a605399a11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 501de8d019da4b8c91035dafd82c6464, 85c4fff9f22d4c1aa35e67a605399a11; no voters:
I20260925 04:40:34.084882 12137 raft_consensus.cc:2781] T 39f8ec4d909b49f0b51dfd86ebaf589f P 85c4fff9f22d4c1aa35e67a605399a11 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:40:34.085139 12137 raft_consensus.cc:686] T 39f8ec4d909b49f0b51dfd86ebaf589f P 85c4fff9f22d4c1aa35e67a605399a11 [term 1 LEADER]: Becoming Leader. State: Replica: 85c4fff9f22d4c1aa35e67a605399a11, State: Running, Role: LEADER
I20260925 04:40:34.085549 12137 consensus_queue.cc:237] T 39f8ec4d909b49f0b51dfd86ebaf589f P 85c4fff9f22d4c1aa35e67a605399a11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85c4fff9f22d4c1aa35e67a605399a11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44939 } } peers { permanent_uuid: "b0f6866121ef4939b5c5139bc8165b63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37607 } } peers { permanent_uuid: "501de8d019da4b8c91035dafd82c6464" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46183 } }
I20260925 04:40:34.089457 11628 catalog_manager.cc:5495] T 39f8ec4d909b49f0b51dfd86ebaf589f P 85c4fff9f22d4c1aa35e67a605399a11 reported cstate change: term changed from 0 to 1, leader changed from <none> to 85c4fff9f22d4c1aa35e67a605399a11 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "85c4fff9f22d4c1aa35e67a605399a11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85c4fff9f22d4c1aa35e67a605399a11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b0f6866121ef4939b5c5139bc8165b63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "501de8d019da4b8c91035dafd82c6464" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46183 } health_report { overall_health: UNKNOWN } } }
W20260925 04:40:34.103593 12113 tablet.cc:2369] T 39f8ec4d909b49f0b51dfd86ebaf589f P 501de8d019da4b8c91035dafd82c6464: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 04:40:34.123152 11843 tablet.cc:2369] T 39f8ec4d909b49f0b51dfd86ebaf589f P 85c4fff9f22d4c1aa35e67a605399a11: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:40:34.132845 12125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:40:34.127393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60360 (local address 127.0.98.1:44939)
0925 04:40:34.127531 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 04:40:34.127535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:40:34.127548 (+ 13us) server_negotiation.cc:407] Connection header received
0925 04:40:34.127662 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:40:34.127666 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:40:34.127751 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:40:34.127857 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:40:34.128261 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:40:34.129068 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:40:34.130379 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:40:34.130728 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:40:34.130836 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:40:34.131280 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:40:34.131936 (+ 656us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:40:34.132072 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 04:40:34.132581 (+ 509us) server_negotiation.cc:299] Negotiation successful
0925 04:40:34.132659 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 04:40:34.141137 11921 raft_consensus.cc:1261] T 39f8ec4d909b49f0b51dfd86ebaf589f P b0f6866121ef4939b5c5139bc8165b63 [term 1 FOLLOWER]: Refusing update from remote peer 85c4fff9f22d4c1aa35e67a605399a11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:40:34.141137 12050 raft_consensus.cc:1261] T 39f8ec4d909b49f0b51dfd86ebaf589f P 501de8d019da4b8c91035dafd82c6464 [term 1 FOLLOWER]: Refusing update from remote peer 85c4fff9f22d4c1aa35e67a605399a11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:40:34.142133 12137 consensus_queue.cc:1035] T 39f8ec4d909b49f0b51dfd86ebaf589f P 85c4fff9f22d4c1aa35e67a605399a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "501de8d019da4b8c91035dafd82c6464" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:40:34.142421 12141 consensus_queue.cc:1035] T 39f8ec4d909b49f0b51dfd86ebaf589f P 85c4fff9f22d4c1aa35e67a605399a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0f6866121ef4939b5c5139bc8165b63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:40:34.153347 12144 mvcc.cc:205] Tried to move back new op lower bound from 7333114815032729600 to 7333114814817640448. Current Snapshot: MvccSnapshot[applied={T|T < 7333114815032729600}]
I20260925 04:40:34.159163 12145 mvcc.cc:205] Tried to move back new op lower bound from 7333114815032729600 to 7333114814817640448. Current Snapshot: MvccSnapshot[applied={T|T < 7333114815032729600}]
I20260925 04:40:34.161494 12146 mvcc.cc:205] Tried to move back new op lower bound from 7333114815032729600 to 7333114814817640448. Current Snapshot: MvccSnapshot[applied={T|T < 7333114815032729600}]
I20260925 04:40:34.172436 11627 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:40:34.172636 11627 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:40:34.175757 11627 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 933b6d73e23c4f259fefbbb50772f11c: Sending DeleteTablet for 3 replicas of tablet 39f8ec4d909b49f0b51dfd86ebaf589f
I20260925 04:40:34.176931 11912 tablet_service.cc:1514] Processing DeleteTablet for tablet 39f8ec4d909b49f0b51dfd86ebaf589f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:40:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33730
I20260925 04:40:34.177000 12046 tablet_service.cc:1514] Processing DeleteTablet for tablet 39f8ec4d909b49f0b51dfd86ebaf589f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:40:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42706
I20260925 04:40:34.177423 11776 tablet_service.cc:1514] Processing DeleteTablet for tablet 39f8ec4d909b49f0b51dfd86ebaf589f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:40:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60348
I20260925 04:40:34.177927 12155 tablet_replica.cc:330] T 39f8ec4d909b49f0b51dfd86ebaf589f P b0f6866121ef4939b5c5139bc8165b63: stopping tablet replica
I20260925 04:40:34.178287 12155 raft_consensus.cc:2229] T 39f8ec4d909b49f0b51dfd86ebaf589f P b0f6866121ef4939b5c5139bc8165b63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:40:34.178555 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 11709
I20260925 04:40:34.179988 12155 raft_consensus.cc:2258] T 39f8ec4d909b49f0b51dfd86ebaf589f P b0f6866121ef4939b5c5139bc8165b63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:40:34.182628 12155 ts_tablet_manager.cc:1905] T 39f8ec4d909b49f0b51dfd86ebaf589f P b0f6866121ef4939b5c5139bc8165b63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:40:34.186161 12155 ts_tablet_manager.cc:1918] T 39f8ec4d909b49f0b51dfd86ebaf589f P b0f6866121ef4939b5c5139bc8165b63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:40:34.186396 12155 log.cc:1196] T 39f8ec4d909b49f0b51dfd86ebaf589f P b0f6866121ef4939b5c5139bc8165b63: Deleting WAL directory at /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/ts-1/wal/wals/39f8ec4d909b49f0b51dfd86ebaf589f
I20260925 04:40:34.186793 12155 ts_tablet_manager.cc:1939] T 39f8ec4d909b49f0b51dfd86ebaf589f P b0f6866121ef4939b5c5139bc8165b63: Deleting consensus metadata
I20260925 04:40:34.187232 12156 tablet_replica.cc:330] T 39f8ec4d909b49f0b51dfd86ebaf589f P 501de8d019da4b8c91035dafd82c6464: stopping tablet replica
I20260925 04:40:34.187597 12156 raft_consensus.cc:2229] T 39f8ec4d909b49f0b51dfd86ebaf589f P 501de8d019da4b8c91035dafd82c6464 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:40:34.188001 12156 raft_consensus.cc:2258] T 39f8ec4d909b49f0b51dfd86ebaf589f P 501de8d019da4b8c91035dafd82c6464 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 04:40:34.188436 11613 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44939 (error 108)
I20260925 04:40:34.188643 11614 catalog_manager.cc:4841] TS b0f6866121ef4939b5c5139bc8165b63 (127.0.98.2:37607): tablet 39f8ec4d909b49f0b51dfd86ebaf589f (table test-table [id=e3b16ad74b86458bbf08200852414aad]) successfully deleted
I20260925 04:40:34.190079 12126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:40:34.189697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44939 (local address 127.0.0.1:60366)
0925 04:40:34.189911 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0925 04:40:34.189987 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
W20260925 04:40:34.190390 11613 catalog_manager.cc:4584] TS 85c4fff9f22d4c1aa35e67a605399a11 (127.0.98.1:44939): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 39f8ec4d909b49f0b51dfd86ebaf589f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44939: connect: Connection refused (error 111)
I20260925 04:40:34.191900 12156 ts_tablet_manager.cc:1905] T 39f8ec4d909b49f0b51dfd86ebaf589f P 501de8d019da4b8c91035dafd82c6464: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:40:34.193538 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 11848
I20260925 04:40:34.195961 12156 ts_tablet_manager.cc:1918] T 39f8ec4d909b49f0b51dfd86ebaf589f P 501de8d019da4b8c91035dafd82c6464: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:40:34.196156 12156 log.cc:1196] T 39f8ec4d909b49f0b51dfd86ebaf589f P 501de8d019da4b8c91035dafd82c6464: Deleting WAL directory at /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311123478858-392-0/minicluster-data/ts-2/wal/wals/39f8ec4d909b49f0b51dfd86ebaf589f
I20260925 04:40:34.196593 12156 ts_tablet_manager.cc:1939] T 39f8ec4d909b49f0b51dfd86ebaf589f P 501de8d019da4b8c91035dafd82c6464: Deleting consensus metadata
I20260925 04:40:34.198258 11612 catalog_manager.cc:4841] TS 501de8d019da4b8c91035dafd82c6464 (127.0.98.3:46183): tablet 39f8ec4d909b49f0b51dfd86ebaf589f (table test-table [id=e3b16ad74b86458bbf08200852414aad]) successfully deleted
I20260925 04:40:34.202508 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 11983
Sep 25 04:40:34 dist-test-slave-dc95 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311233, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 04:40:34.210269 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 11596
I20260925 04:40:34.216853 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 04:40:34.552567 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 04:40:34.552688 392 mini_ranger.cc:66] Stopping Ranger...
I20260925 04:40:34.888588 392 mini_ranger.cc:68] Stopped Ranger
2026-09-25 04:40:34.888 UTC [11182] LOG: received smart shutdown request
2026-09-25 04:40:34.892 UTC [11182] LOG: background worker "logical replication launcher" (PID 11191) exited with exit code 1
2026-09-25 04:40:34.892 UTC [11186] LOG: shutting down
2026-09-25 04:40:35.103 UTC [11182] LOG: database system is shut down
2026-09-25T04:40:35Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47281 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:40:35 dist-test-slave-dc95 krb5kdc[12168](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:40:35 dist-test-slave-dc95 krb5kdc[12168](info): set up 2 sockets
Sep 25 04:40:35 dist-test-slave-dc95 krb5kdc[12168](info): commencing operation
krb5kdc: starting...
W20260925 04:40:36.167634 392 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/krb5kdc/test-user.keytab.
Sep 25 04:40:36 dist-test-slave-dc95 krb5kdc[12168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311236, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:40:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:40:36Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260925 04:40:36.348394 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskilA2AO/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/postgres -l logfile start
2026-09-25 04:40:39.030 UTC [12198] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 04:40:39.030 UTC [12198] LOG: listening on IPv4 address "127.0.98.20", port 36959
2026-09-25 04:40:39.034 UTC [12198] LOG: listening on Unix socket "/tmp/.s.PGSQL.36959"
2026-09-25 04:40:39.062 UTC [12201] LOG: database system was shut down at 2026-09-25 04:40:37 UTC
2026-09-25 04:40:39.067 UTC [12198] LOG: database system is ready to accept connections
I20260925 04:40:39.927388 392 mini_postgres.cc:96] Postgres bound to 36959
2026-09-25 04:40:39.933 UTC [12209] FATAL: database "slave" does not exist
127.0.98.20:36959 - accepting connections
I20260925 04:40:39.934079 392 mini_ranger.cc:161] Starting Ranger...
I20260925 04:40:39.949005 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 04:40:40.485666 392 mini_ranger.cc:87] Created ranger Postgres database
I20260925 04:40:40.485743 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ranger-admin
2026-09-25 04:40:40,758 [I] DB FLAVOR :POSTGRES
2026-09-25 04:40:40,758 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 04:40:40,758 [I] Checking connection..
2026-09-25 04:40:40,758 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskilA2AO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskilA2AO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 04:40:41,017 [I] Checking connection passed.
2026-09-25 04:40:41,017 [I] --------- Verifying version history table ---------
2026-09-25 04:40:41,017 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskilA2AO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskilA2AO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 04:40:41,286 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 04:40:41,287 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskilA2AO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskilA2AO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 04:40:41,554 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 04:40:41,555 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 04:40:41,555 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskilA2AO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskilA2AO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36959/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskilA2AO/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 04:40:41.828 UTC [12273] WARNING: there is no transaction in progress
2026-09-25 04:40:41,834 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 04:40:41,835 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskilA2AO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskilA2AO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 04:40:42,092 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 04:40:42,092 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 04:40:42,092 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskilA2AO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskilA2AO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 04:40:42,331 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskilA2AO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskilA2AO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 04:40:42,577 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskilA2AO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskilA2AO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-dc95.c.gcp-upstream.internal','N') ;"
2026-09-25 04:40:42,818 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskilA2AO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskilA2AO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 04:40:43,059 [I] Table x_portal_user does not exist in database ranger
2026-09-25 04:40:43,059 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskilA2AO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskilA2AO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 04:40:43,307 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 04:40:43,307 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskilA2AO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskilA2AO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 04:40:43,561 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskilA2AO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskilA2AO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 04:40:43,825 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 04:40:43,826 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskilA2AO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskilA2AO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36959/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskilA2AO/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 04:40:44.903 UTC [12381] WARNING: there is no transaction in progress
2026-09-25 04:40:44.922 UTC [12381] WARNING: there is no transaction in progress
2026-09-25 04:40:44.940 UTC [12381] WARNING: there is no transaction in progress
2026-09-25 04:40:44.958 UTC [12381] WARNING: there is no transaction in progress
2026-09-25 04:40:44.977 UTC [12381] WARNING: there is no transaction in progress
2026-09-25 04:40:44.998 UTC [12381] WARNING: there is no transaction in progress
2026-09-25 04:40:45.068 UTC [12381] WARNING: there is no transaction in progress
2026-09-25 04:40:45.076 UTC [12381] WARNING: there is no transaction in progress
2026-09-25 04:40:45.093 UTC [12381] WARNING: there is no transaction in progress
2026-09-25 04:40:45.105 UTC [12381] WARNING: there is no transaction in progress
2026-09-25 04:40:45.118 UTC [12381] WARNING: there is no transaction in progress
2026-09-25 04:40:45.133 UTC [12381] WARNING: there is no transaction in progress
2026-09-25 04:40:45.144 UTC [12381] WARNING: there is no transaction in progress
2026-09-25 04:40:45.152 UTC [12381] WARNING: there is no transaction in progress
2026-09-25 04:40:45.160 UTC [12381] WARNING: there is no transaction in progress
2026-09-25 04:40:45.169 UTC [12381] WARNING: there is no transaction in progress
2026-09-25 04:40:45,948 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 04:40:45,948 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskilA2AO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskilA2AO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 04:40:46,198 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 04:40:46,198 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskilA2AO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskilA2AO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 04:40:46,460 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 04:40:46,461 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskilA2AO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskilA2AO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 04:40:46,699 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 04:40:46,700 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskilA2AO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskilA2AO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 04:40:46,946 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskilA2AO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskilA2AO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 04:40:47,213 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskilA2AO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskilA2AO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-dc95.c.gcp-upstream.internal';"
2026-09-25 04:40:47,470 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 04:40:47,471 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskilA2AO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskilA2AO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 04:40:47,743 [I] DB_PATCHES have already been applied
I20260925 04:40:47.748929 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskilA2AO/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskilA2AO/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskilA2AO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskilA2AO/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 04:40:47.749012 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260925 04:40:47.752542 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:48849
Sep 25, 2026 4:40:48 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 4:40:48 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 4:40:48 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 4:40:48 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 4:40:48 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 4:40:48 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 4:40:48 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 04:40:49.762759 392 mini_ranger.cc:160] Time spent starting Ranger: real 9.829s user 0.005s sys 0.006s
Sep 25, 2026 4:41:09 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 4:41:09 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 25, 2026 4:41:09 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 4:41:09 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 4:41:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 4:41:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 4:41:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 4:41:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 4:41:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 4:41:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 4:41:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 4:41:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 4:41:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 4:41:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 4:41:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 4:41:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 4:41:09 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 4:41:10 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 04:41:11.298863 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 04:41:11 dist-test-slave-dc95 krb5kdc[12168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311271, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 04:41:11.402976 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 04:41:11.413690 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 04:41:12.318486 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 04:41:12.318575 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ranger-kms
I20260925 04:41:12.318593 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:36959
2026-09-25 04:41:12,479 [I] DB FLAVOR :POSTGRES
2026-09-25 04:41:12,479 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 04:41:12,480 [I] Checking connection
2026-09-25 04:41:12,480 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskilA2AO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36959/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 04:41:12,739 [I] connection success
2026-09-25 04:41:12,739 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 04:41:12,739 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 04:41:12,739 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskilA2AO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36959/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 04:41:13,020 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 04:41:13,020 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 04:41:13,020 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 04:41:13,020 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskilA2AO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36959/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 04:41:13,355 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 04:41:13.866436 392 mini_ranger_kms.cc:310] Created kms service
I20260925 04:41:14.047179 392 mini_ranger_kms.cc:326] Created kudu user
I20260925 04:41:14.092741 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 04:41:14 dist-test-slave-dc95 krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311271, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 04:41:14.499889 392 mini_ranger_kms.cc:384] Added ranger policy
I20260925 04:41:14.500062 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskilA2AO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskilA2AO/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskilA2AO/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 04:41:14.500095 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260925 04:41:14.505020 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12563
I20260925 04:41:14.505115 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:49397
Sep 25, 2026 4:41:15 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 4:41:15 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 4:41:15 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 4:41:15 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 4:41:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 4:41:16 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 04:41:16.525012 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.122s user 0.026s sys 0.118s
I20260925 04:41:16.525125 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 04:41:16.525200 392 mini_ranger_kms.cc:401] 127.0.98.20:49397/kms/v1/keys
Sep 25 04:41:16 dist-test-slave-dc95 krb5kdc[12168](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790311276, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 04:41:17 dist-test-slave-dc95 krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311276, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 25 04:41:17 dist-test-slave-dc95 krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311276, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 25 04:41:19 dist-test-slave-dc95 krb5kdc[12168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 04:41:19.888382 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34431
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34431
--ranger_config_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:49397/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:41:20.037891 12613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:41:20.038196 12613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:41:20.038264 12613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:41:20.042608 12613 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:41:20.042701 12613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:41:20.042728 12613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:41:20.042749 12613 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:41:20.042771 12613 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:41:20.049155 12613 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:49397/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34431
--ranger_config_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:41:20.050339 12613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:41:20.051434 12613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:41:20.061956 12621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:41:20.062062 12619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:41:20.061973 12618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:41:20.062184 12613 server_base.cc:1034] running on GCE node
I20260925 04:41:20.062945 12613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:41:20.064181 12613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:41:20.065375 12613 hybrid_clock.cc:648] HybridClock initialized: now 1790311280065350 us; error 42 us; skew 500 ppm
Sep 25 04:41:20 dist-test-slave-dc95 krb5kdc[12168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311280, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:41:20.075327 12613 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 04:41:20.076536 12613 webserver.cc:460] Webserver started at http://127.0.98.62:43593/ using document root <none> and password file <none>
I20260925 04:41:20.077142 12613 fs_manager.cc:359] Metadata directory not provided
I20260925 04:41:20.077206 12613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:41:20.077430 12613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 04:41:20 dist-test-slave-dc95 krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311280, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 04:41:20.142223 12613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/master-0/data/instance:
uuid: "2ebdac1e6f7941dd8754c15320ab8b04"
format_stamp: "Formatted at 2026-09-25 04:41:20 on dist-test-slave-dc95"
server_key: "f2a537adc7c24ffb76ac6660d5e3ec22"
server_key_iv: "e7bde89304cdfabfecf203f872f65812"
server_key_version: "kuduclusterkey@0"
I20260925 04:41:20.142901 12613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/master-0/wal/instance:
uuid: "2ebdac1e6f7941dd8754c15320ab8b04"
format_stamp: "Formatted at 2026-09-25 04:41:20 on dist-test-slave-dc95"
server_key: "f2a537adc7c24ffb76ac6660d5e3ec22"
server_key_iv: "e7bde89304cdfabfecf203f872f65812"
server_key_version: "kuduclusterkey@0"
I20260925 04:41:20.147012 12613 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 04:41:20.165577 12628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:41:20.167268 12613 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:41:20.167488 12613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/master-0/wal
uuid: "2ebdac1e6f7941dd8754c15320ab8b04"
format_stamp: "Formatted at 2026-09-25 04:41:20 on dist-test-slave-dc95"
server_key: "f2a537adc7c24ffb76ac6660d5e3ec22"
server_key_iv: "e7bde89304cdfabfecf203f872f65812"
server_key_version: "kuduclusterkey@0"
I20260925 04:41:20.167666 12613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:41:20.220216 12613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:41:20.223392 12613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:41:20.223652 12613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:41:20.231516 12613 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34431
I20260925 04:41:20.231561 12680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34431 every 8 connection(s)
I20260925 04:41:20.232805 12613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
I20260925 04:41:20.235888 12681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:41:20.238440 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 12613
I20260925 04:41:20.238581 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/master-0/wal/instance
I20260925 04:41:20.238874 392 external_mini_cluster.cc:1428] Setting key d88f1d87ede865d15c864c4affc9c608
I20260925 04:41:20.242767 12681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ebdac1e6f7941dd8754c15320ab8b04: Bootstrap starting.
I20260925 04:41:20.245844 12681 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ebdac1e6f7941dd8754c15320ab8b04: Neither blocks nor log segments found. Creating new log.
I20260925 04:41:20.246722 12681 log.cc:824] T 00000000000000000000000000000000 P 2ebdac1e6f7941dd8754c15320ab8b04: Log is configured to *not* fsync() on all Append() calls
I20260925 04:41:20.249433 12681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ebdac1e6f7941dd8754c15320ab8b04: No bootstrap required, opened a new log
Sep 25 04:41:20 dist-test-slave-dc95 krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:41:20.254019 12681 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ebdac1e6f7941dd8754c15320ab8b04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ebdac1e6f7941dd8754c15320ab8b04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34431 } }
I20260925 04:41:20.254470 12681 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ebdac1e6f7941dd8754c15320ab8b04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:41:20.254580 12681 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ebdac1e6f7941dd8754c15320ab8b04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ebdac1e6f7941dd8754c15320ab8b04, State: Initialized, Role: FOLLOWER
I20260925 04:41:20.255158 12681 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ebdac1e6f7941dd8754c15320ab8b04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ebdac1e6f7941dd8754c15320ab8b04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34431 } }
I20260925 04:41:20.255347 12681 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ebdac1e6f7941dd8754c15320ab8b04 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:41:20.255455 12681 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ebdac1e6f7941dd8754c15320ab8b04 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:41:20.255666 12681 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ebdac1e6f7941dd8754c15320ab8b04 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:41:20.257382 12681 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ebdac1e6f7941dd8754c15320ab8b04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ebdac1e6f7941dd8754c15320ab8b04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34431 } }
I20260925 04:41:20.257800 12681 leader_election.cc:302] T 00000000000000000000000000000000 P 2ebdac1e6f7941dd8754c15320ab8b04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ebdac1e6f7941dd8754c15320ab8b04; no voters:
I20260925 04:41:20.258004 12684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:20.240512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56338 (local address 127.0.98.62:34431)
0925 04:41:20.240994 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0925 04:41:20.241013 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 04:41:20.241056 (+ 43us) server_negotiation.cc:407] Connection header received
0925 04:41:20.241915 (+ 859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:41:20.241944 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:41:20.242420 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:41:20.242895 (+ 475us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:41:20.243764 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:20.245238 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:20.246472 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:20.247177 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:20.247356 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:20.251138 (+ 3782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:41:20.251153 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:41:20.251176 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:41:20.251216 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:41:20.253449 (+ 2233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:20.254310 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:20.254317 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:20.254324 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:20.254485 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:20.254866 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:20.254869 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:20.254871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:20.255425 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:41:20.255678 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0925 04:41:20.255996 (+ 318us) server_negotiation.cc:299] Negotiation successful
0925 04:41:20.256942 (+ 946us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":139,"threads_started":1}
I20260925 04:41:20.258198 12681 leader_election.cc:288] T 00000000000000000000000000000000 P 2ebdac1e6f7941dd8754c15320ab8b04 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:41:20.259573 12681 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ebdac1e6f7941dd8754c15320ab8b04 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:41:20.261869 12681 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 04:41:20.262946 12686 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ebdac1e6f7941dd8754c15320ab8b04 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:41:20.265323 12686 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ebdac1e6f7941dd8754c15320ab8b04 [term 1 LEADER]: Becoming Leader. State: Replica: 2ebdac1e6f7941dd8754c15320ab8b04, State: Running, Role: LEADER
I20260925 04:41:20.265964 12686 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ebdac1e6f7941dd8754c15320ab8b04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ebdac1e6f7941dd8754c15320ab8b04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34431 } }
I20260925 04:41:20.268286 12687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ebdac1e6f7941dd8754c15320ab8b04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ebdac1e6f7941dd8754c15320ab8b04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ebdac1e6f7941dd8754c15320ab8b04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34431 } } }
I20260925 04:41:20.268472 12687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ebdac1e6f7941dd8754c15320ab8b04 [sys.catalog]: This master's current role is: LEADER
I20260925 04:41:20.268282 12700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ebdac1e6f7941dd8754c15320ab8b04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ebdac1e6f7941dd8754c15320ab8b04. Latest consensus state: current_term: 1 leader_uuid: "2ebdac1e6f7941dd8754c15320ab8b04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ebdac1e6f7941dd8754c15320ab8b04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34431 } } }
I20260925 04:41:20.268630 12700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ebdac1e6f7941dd8754c15320ab8b04 [sys.catalog]: This master's current role is: LEADER
I20260925 04:41:20.268999 12703 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:41:20.273326 12703 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:41:20.280820 12703 catalog_manager.cc:1347] Generated new cluster ID: 3c8f1368b8ea47f68d5de23e2f91208f
I20260925 04:41:20.281010 12703 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:41:20.296355 12703 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:41:20.297942 12703 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:41:20.305830 12703 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ebdac1e6f7941dd8754c15320ab8b04: Generated new TSK 0
I20260925 04:41:20.306926 12703 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 04:41:21.035858 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:49397/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34431
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:41:21.180037 12726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:41:21.180408 12726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:41:21.180476 12726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:41:21.186157 12726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:41:21.186245 12726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:41:21.186311 12726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 04:41:21.192730 12726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:49397/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:41:21.193943 12726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:41:21.195009 12726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:41:21.208081 12734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:41:21.208110 12733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:41:21.208081 12736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:41:21.208698 12726 server_base.cc:1034] running on GCE node
I20260925 04:41:21.209137 12726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:41:21.209890 12726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:41:21.211125 12726 hybrid_clock.cc:648] HybridClock initialized: now 1790311281211093 us; error 43 us; skew 500 ppm
Sep 25 04:41:21 dist-test-slave-dc95 krb5kdc[12168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311281, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:41:21.214228 12726 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 04:41:21.215179 12726 webserver.cc:460] Webserver started at http://127.0.98.1:39161/ using document root <none> and password file <none>
I20260925 04:41:21.215837 12726 fs_manager.cc:359] Metadata directory not provided
I20260925 04:41:21.215912 12726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:41:21.216140 12726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 04:41:21 dist-test-slave-dc95 krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311281, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 04:41:21.233758 12726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-0/data/instance:
uuid: "76ebc568de434d39b3f727e77ed7faea"
format_stamp: "Formatted at 2026-09-25 04:41:21 on dist-test-slave-dc95"
server_key: "fc87a223179576442ea0400a9b3b4d3f"
server_key_iv: "9bdc67cad8b38c3314c246e0d76e2d35"
server_key_version: "kuduclusterkey@0"
I20260925 04:41:21.236214 12726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-0/wal/instance:
uuid: "76ebc568de434d39b3f727e77ed7faea"
format_stamp: "Formatted at 2026-09-25 04:41:21 on dist-test-slave-dc95"
server_key: "fc87a223179576442ea0400a9b3b4d3f"
server_key_iv: "9bdc67cad8b38c3314c246e0d76e2d35"
server_key_version: "kuduclusterkey@0"
I20260925 04:41:21.245568 12726 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.004s sys 0.000s
I20260925 04:41:21.266693 12743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:41:21.268806 12726 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 04:41:21.269083 12726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-0/wal
uuid: "76ebc568de434d39b3f727e77ed7faea"
format_stamp: "Formatted at 2026-09-25 04:41:21 on dist-test-slave-dc95"
server_key: "fc87a223179576442ea0400a9b3b4d3f"
server_key_iv: "9bdc67cad8b38c3314c246e0d76e2d35"
server_key_version: "kuduclusterkey@0"
I20260925 04:41:21.269368 12726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:41:21.291119 12726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:41:21.294981 12726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:41:21.295259 12726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:41:21.296131 12726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:41:21.297312 12726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:41:21.297433 12726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:41:21.297557 12726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:41:21.297636 12726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:41:21.311269 12726 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38785
I20260925 04:41:21.311304 12857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38785 every 8 connection(s)
I20260925 04:41:21.312501 12726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
I20260925 04:41:21.315014 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 12726
I20260925 04:41:21.315131 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-0/wal/instance
I20260925 04:41:21.315446 392 external_mini_cluster.cc:1428] Setting key d6ad88093dbf5c6e048a6a20b1116715
Sep 25 04:41:21 dist-test-slave-dc95 krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311281, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 04:41:21.329717 12861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:21.314541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60483 (local address 127.0.98.62:34431)
0925 04:41:21.314860 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0925 04:41:21.314865 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:41:21.316550 (+ 1685us) server_negotiation.cc:407] Connection header received
0925 04:41:21.317789 (+ 1239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:41:21.317793 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:41:21.317847 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:41:21.317936 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:41:21.319483 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:21.320345 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:21.321685 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:21.322004 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:21.322111 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:21.325312 (+ 3201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:41:21.325317 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:41:21.325320 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:41:21.325351 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:41:21.327212 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:21.327997 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:21.328002 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:21.328004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:21.328108 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:21.328468 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:21.328471 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:21.328473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:21.328670 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:41:21.328753 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 04:41:21.329399 (+ 646us) server_negotiation.cc:299] Negotiation successful
0925 04:41:21.329531 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":89,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260925 04:41:21.330755 12862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:21.314984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34431 (local address 127.0.98.1:60483)
0925 04:41:21.315587 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:21.315676 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0925 04:41:21.317538 (+ 1862us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:41:21.318279 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:41:21.318289 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:41:21.318645 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:41:21.319277 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:21.319293 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:21.320603 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:21.320607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:21.321556 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:21.321566 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:21.322302 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:21.322305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:21.322509 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:21.323234 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:41:21.323256 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:41:21.325082 (+ 1826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:41:21.327366 (+ 2284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:41:21.327373 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:41:21.327395 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:41:21.327871 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:41:21.328193 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:41:21.328196 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:41:21.328198 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:41:21.328358 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:41:21.328779 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:41:21.328787 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:41:21.329136 (+ 349us) client_negotiation.cc:769] Sending connection context
0925 04:41:21.329326 (+ 190us) client_negotiation.cc:240] Negotiation successful
0925 04:41:21.329677 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":155,"threads_started":1}
I20260925 04:41:21.332420 12859 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34431
I20260925 04:41:21.332674 12859 heartbeater.cc:464] Registering TS with master...
I20260925 04:41:21.333279 12859 heartbeater.cc:511] Master 127.0.98.62:34431 requested a full tablet report, sending...
I20260925 04:41:21.334990 12644 ts_manager.cc:194] Registered new tserver with Master: 76ebc568de434d39b3f727e77ed7faea (127.0.98.1:38785)
I20260925 04:41:21.336858 12644 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60483
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 04:41:21.352101 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:49397/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34431
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 04:41:21 dist-test-slave-dc95 krb5kdc[12168](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790311281, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 04:41:21.517755 12865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:41:21.518265 12865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:41:21.518383 12865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:41:21.524731 12865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:41:21.524875 12865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:41:21.524999 12865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 04:41:21.532240 12865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:49397/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:41:21.533818 12865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:41:21.535233 12865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:41:21.548331 12874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:41:21.548512 12872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:41:21.548825 12871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:41:21.549062 12865 server_base.cc:1034] running on GCE node
I20260925 04:41:21.549484 12865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:41:21.550109 12865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:41:21.551295 12865 hybrid_clock.cc:648] HybridClock initialized: now 1790311281551266 us; error 40 us; skew 500 ppm
Sep 25 04:41:21 dist-test-slave-dc95 krb5kdc[12168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311281, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:41:21.557407 12865 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 04:41:21.558430 12865 webserver.cc:460] Webserver started at http://127.0.98.2:42389/ using document root <none> and password file <none>
I20260925 04:41:21.558996 12865 fs_manager.cc:359] Metadata directory not provided
I20260925 04:41:21.559062 12865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:41:21.559283 12865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 04:41:21 dist-test-slave-dc95 krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311281, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 04:41:21.578953 12865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-1/data/instance:
uuid: "8de887c7a4c1454384a914454b9e6457"
format_stamp: "Formatted at 2026-09-25 04:41:21 on dist-test-slave-dc95"
server_key: "c0bc54f554ed32f4d7c931f86594447b"
server_key_iv: "d8c319efe95b4359c43d2c59d7295a28"
server_key_version: "kuduclusterkey@0"
I20260925 04:41:21.579708 12865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8de887c7a4c1454384a914454b9e6457"
format_stamp: "Formatted at 2026-09-25 04:41:21 on dist-test-slave-dc95"
server_key: "c0bc54f554ed32f4d7c931f86594447b"
server_key_iv: "d8c319efe95b4359c43d2c59d7295a28"
server_key_version: "kuduclusterkey@0"
I20260925 04:41:21.584079 12865 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 04:41:21.601564 12881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:41:21.603111 12865 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:41:21.603251 12865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-1/wal
uuid: "8de887c7a4c1454384a914454b9e6457"
format_stamp: "Formatted at 2026-09-25 04:41:21 on dist-test-slave-dc95"
server_key: "c0bc54f554ed32f4d7c931f86594447b"
server_key_iv: "d8c319efe95b4359c43d2c59d7295a28"
server_key_version: "kuduclusterkey@0"
I20260925 04:41:21.603380 12865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:41:21.643496 12865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:41:21.646479 12865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:41:21.646668 12865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:41:21.647298 12865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:41:21.648209 12865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:41:21.648257 12865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:41:21.648309 12865 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:41:21.648334 12865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:41:21.657394 12865 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34677
I20260925 04:41:21.657449 12994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34677 every 8 connection(s)
I20260925 04:41:21.658372 12865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
I20260925 04:41:21.663000 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 12865
I20260925 04:41:21.663136 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-1/wal/instance
I20260925 04:41:21.663383 392 external_mini_cluster.cc:1428] Setting key ea967edf7ec718defde31bd24fbe6e51
Sep 25 04:41:21 dist-test-slave-dc95 krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311281, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260925 04:41:21.675241 12861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:21.660486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43367 (local address 127.0.98.62:34431)
0925 04:41:21.660660 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 04:41:21.660665 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:41:21.661662 (+ 997us) server_negotiation.cc:407] Connection header received
0925 04:41:21.662982 (+ 1320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:41:21.662986 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:41:21.663047 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:41:21.663100 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:41:21.664773 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:21.665505 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:21.667271 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:21.667498 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:21.667577 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:21.670640 (+ 3063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:41:21.670644 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:41:21.670647 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:41:21.670682 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:41:21.672581 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:21.673340 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:21.673343 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:21.673344 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:21.673425 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:21.673927 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:21.673931 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:21.673933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:21.674143 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:41:21.674220 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 04:41:21.674967 (+ 747us) server_negotiation.cc:299] Negotiation successful
0925 04:41:21.675114 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260925 04:41:21.682606 12997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:21.660927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34431 (local address 127.0.98.2:43367)
0925 04:41:21.661484 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:21.661548 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0925 04:41:21.662707 (+ 1159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:41:21.663532 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:41:21.663542 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:41:21.663925 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:41:21.664630 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:21.664650 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:21.665657 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:21.665662 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:21.667152 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:21.667166 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:21.667681 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:21.667686 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:21.667882 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:21.668597 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:41:21.668617 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:41:21.670422 (+ 1805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:41:21.672766 (+ 2344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:41:21.672775 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:41:21.672798 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:41:21.673212 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:41:21.673528 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:41:21.673531 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:41:21.673533 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:41:21.673801 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:41:21.674250 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:41:21.674258 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:41:21.674667 (+ 409us) client_negotiation.cc:769] Sending connection context
0925 04:41:21.674945 (+ 278us) client_negotiation.cc:240] Negotiation successful
0925 04:41:21.675338 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":104,"threads_started":1}
I20260925 04:41:21.684268 12995 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34431
I20260925 04:41:21.684581 12995 heartbeater.cc:464] Registering TS with master...
I20260925 04:41:21.685253 12995 heartbeater.cc:511] Master 127.0.98.62:34431 requested a full tablet report, sending...
I20260925 04:41:21.686611 12644 ts_manager.cc:194] Registered new tserver with Master: 8de887c7a4c1454384a914454b9e6457 (127.0.98.2:34677)
I20260925 04:41:21.687407 12644 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43367
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 04:41:21.711054 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:49397/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34431
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 04:41:21 dist-test-slave-dc95 krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311281, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260925 04:41:21.884223 13000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:41:21.884639 13000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:41:21.884711 13000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:41:21.890779 13000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:41:21.890887 13000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:41:21.890971 13000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 04:41:21.899529 13000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:49397/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:41:21.901041 13000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:41:21.902457 13000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:41:21.914522 13008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:41:21.914605 13006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:41:21.914736 13000 server_base.cc:1034] running on GCE node
W20260925 04:41:21.914582 13005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:41:21.915336 13000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:41:21.916105 13000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:41:21.917295 13000 hybrid_clock.cc:648] HybridClock initialized: now 1790311281917261 us; error 54 us; skew 500 ppm
Sep 25 04:41:21 dist-test-slave-dc95 krb5kdc[12168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311281, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:41:21.920123 13000 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 04:41:21.921135 13000 webserver.cc:460] Webserver started at http://127.0.98.3:39377/ using document root <none> and password file <none>
I20260925 04:41:21.921712 13000 fs_manager.cc:359] Metadata directory not provided
I20260925 04:41:21.921777 13000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:41:21.921972 13000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 04:41:21 dist-test-slave-dc95 krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311281, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 04:41:21.937557 13000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-2/data/instance:
uuid: "d622c3c03bcf4214904aa23c1ebd7ff9"
format_stamp: "Formatted at 2026-09-25 04:41:21 on dist-test-slave-dc95"
server_key: "d6cb89a9e0fbb66d5c7ff42c4d5a98b2"
server_key_iv: "d425b4a83c5167d893a94a961e307f68"
server_key_version: "kuduclusterkey@0"
I20260925 04:41:21.938817 13000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d622c3c03bcf4214904aa23c1ebd7ff9"
format_stamp: "Formatted at 2026-09-25 04:41:21 on dist-test-slave-dc95"
server_key: "d6cb89a9e0fbb66d5c7ff42c4d5a98b2"
server_key_iv: "d425b4a83c5167d893a94a961e307f68"
server_key_version: "kuduclusterkey@0"
I20260925 04:41:21.943161 13000 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 04:41:21.962772 13015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:41:21.964565 13000 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.004s sys 0.000s
I20260925 04:41:21.964704 13000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-2/wal
uuid: "d622c3c03bcf4214904aa23c1ebd7ff9"
format_stamp: "Formatted at 2026-09-25 04:41:21 on dist-test-slave-dc95"
server_key: "d6cb89a9e0fbb66d5c7ff42c4d5a98b2"
server_key_iv: "d425b4a83c5167d893a94a961e307f68"
server_key_version: "kuduclusterkey@0"
I20260925 04:41:21.964833 13000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:41:21.986860 13000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:41:21.990588 13000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:41:21.990778 13000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:41:21.991434 13000 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:41:21.992522 13000 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:41:21.992599 13000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:41:21.992651 13000 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:41:21.992678 13000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:41:22.003502 13000 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33591
I20260925 04:41:22.004050 13128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33591 every 8 connection(s)
I20260925 04:41:22.004698 13000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
I20260925 04:41:22.013334 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 13000
I20260925 04:41:22.013463 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-2/wal/instance
I20260925 04:41:22.013737 392 external_mini_cluster.cc:1428] Setting key fce1a383cad19c477655de066770b298
Sep 25 04:41:22 dist-test-slave-dc95 krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311281, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:41:22.022619 12861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:22.006836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49877 (local address 127.0.98.62:34431)
0925 04:41:22.006981 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 04:41:22.006986 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:41:22.008542 (+ 1556us) server_negotiation.cc:407] Connection header received
0925 04:41:22.009771 (+ 1229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:41:22.009775 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:41:22.009833 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:41:22.009920 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:41:22.011464 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:22.012327 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:22.014382 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:22.014732 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:22.014816 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:22.018079 (+ 3263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:41:22.018084 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:41:22.018087 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:41:22.018130 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:41:22.019886 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:22.020611 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:22.020615 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:22.020617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:22.020719 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:22.021209 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:22.021213 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:22.021215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:22.021472 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:41:22.021582 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 04:41:22.022285 (+ 703us) server_negotiation.cc:299] Negotiation successful
0925 04:41:22.022440 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 04:41:22.023716 13131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:22.007358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34431 (local address 127.0.98.3:49877)
0925 04:41:22.008355 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:22.008413 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 04:41:22.009526 (+ 1113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:41:22.010142 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:41:22.010152 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:41:22.010530 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:41:22.011263 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:22.011287 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:22.012498 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:22.012502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:22.013983 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:22.013995 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:22.015043 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:22.015047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:22.015290 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:22.016044 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:41:22.016067 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:41:22.017845 (+ 1778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:41:22.020058 (+ 2213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:41:22.020066 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:41:22.020101 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:41:22.020490 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:41:22.020815 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:41:22.020819 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:41:22.020821 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:41:22.021060 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:41:22.021591 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:41:22.021599 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:41:22.021993 (+ 394us) client_negotiation.cc:769] Sending connection context
0925 04:41:22.022221 (+ 228us) client_negotiation.cc:240] Negotiation successful
0925 04:41:22.022594 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":757,"thread_start_us":174,"threads_started":1}
I20260925 04:41:22.025040 13129 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34431
I20260925 04:41:22.025321 13129 heartbeater.cc:464] Registering TS with master...
I20260925 04:41:22.025933 13129 heartbeater.cc:511] Master 127.0.98.62:34431 requested a full tablet report, sending...
I20260925 04:41:22.027402 12645 ts_manager.cc:194] Registered new tserver with Master: d622c3c03bcf4214904aa23c1ebd7ff9 (127.0.98.3:33591)
I20260925 04:41:22.028241 12645 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49877
I20260925 04:41:22.029538 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:41:22.030294 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 12726
I20260925 04:41:22.035491 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 12865
I20260925 04:41:22.041929 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 13000
I20260925 04:41:22.048280 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 12613
I20260925 04:41:22.054440 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34431
--webserver_interface=127.0.98.62
--webserver_port=43593
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34431
--ranger_config_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:49397/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:49397/kms with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:41:22.166388 13132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:41:22.166702 13132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:41:22.166759 13132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:41:22.171057 13132 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:41:22.171141 13132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:41:22.171167 13132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:41:22.171191 13132 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:41:22.171221 13132 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:41:22.177111 13132 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:49397/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34431
--ranger_config_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=43593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:41:22.178270 13132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:41:22.179311 13132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:41:22.188261 13138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:41:22.188233 13140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:41:22.188233 13137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:41:22.188292 13132 server_base.cc:1034] running on GCE node
I20260925 04:41:22.189167 13132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:41:22.190371 13132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:41:22.191579 13132 hybrid_clock.cc:648] HybridClock initialized: now 1790311282191520 us; error 32 us; skew 500 ppm
Sep 25 04:41:22 dist-test-slave-dc95 krb5kdc[12168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311282, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:41:22.194352 13132 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 04:41:22.195287 13132 webserver.cc:460] Webserver started at http://127.0.98.62:43593/ using document root <none> and password file <none>
I20260925 04:41:22.195855 13132 fs_manager.cc:359] Metadata directory not provided
I20260925 04:41:22.195924 13132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 04:41:22 dist-test-slave-dc95 krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311282, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 04:41:22.259809 13132 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:41:22.262264 13148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:41:22.264041 13132 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:41:22.264197 13132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/master-0/wal
uuid: "2ebdac1e6f7941dd8754c15320ab8b04"
format_stamp: "Formatted at 2026-09-25 04:41:20 on dist-test-slave-dc95"
server_key: "f2a537adc7c24ffb76ac6660d5e3ec22"
server_key_iv: "e7bde89304cdfabfecf203f872f65812"
server_key_version: "kuduclusterkey@0"
I20260925 04:41:22.264804 13132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:41:22.282652 13132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:41:22.285521 13132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:41:22.285739 13132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:41:22.293211 13132 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34431
I20260925 04:41:22.293268 13200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34431 every 8 connection(s)
I20260925 04:41:22.294366 13132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
I20260925 04:41:22.298856 13201 sys_catalog.cc:263] Verifying existing consensus state
I20260925 04:41:22.300870 13201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ebdac1e6f7941dd8754c15320ab8b04: Bootstrap starting.
I20260925 04:41:22.302742 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 13132
I20260925 04:41:22.303252 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:49397/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:38785
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34431
--webserver_port=39161
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:49397/kms with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:41:22.318189 13201 log.cc:824] T 00000000000000000000000000000000 P 2ebdac1e6f7941dd8754c15320ab8b04: Log is configured to *not* fsync() on all Append() calls
I20260925 04:41:22.325387 13201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ebdac1e6f7941dd8754c15320ab8b04: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 04:41:22.325802 13201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ebdac1e6f7941dd8754c15320ab8b04: Bootstrap complete.
I20260925 04:41:22.329885 13201 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ebdac1e6f7941dd8754c15320ab8b04 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ebdac1e6f7941dd8754c15320ab8b04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34431 } }
I20260925 04:41:22.330331 13201 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ebdac1e6f7941dd8754c15320ab8b04 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ebdac1e6f7941dd8754c15320ab8b04, State: Initialized, Role: FOLLOWER
I20260925 04:41:22.330829 13201 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ebdac1e6f7941dd8754c15320ab8b04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ebdac1e6f7941dd8754c15320ab8b04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34431 } }
I20260925 04:41:22.330967 13201 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ebdac1e6f7941dd8754c15320ab8b04 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:41:22.331034 13201 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ebdac1e6f7941dd8754c15320ab8b04 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:41:22.331131 13201 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ebdac1e6f7941dd8754c15320ab8b04 [term 1 FOLLOWER]: Advancing to term 2
I20260925 04:41:22.333220 13201 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ebdac1e6f7941dd8754c15320ab8b04 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ebdac1e6f7941dd8754c15320ab8b04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34431 } }
I20260925 04:41:22.333565 13201 leader_election.cc:302] T 00000000000000000000000000000000 P 2ebdac1e6f7941dd8754c15320ab8b04 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ebdac1e6f7941dd8754c15320ab8b04; no voters:
I20260925 04:41:22.333913 13201 leader_election.cc:288] T 00000000000000000000000000000000 P 2ebdac1e6f7941dd8754c15320ab8b04 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 04:41:22.333997 13205 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ebdac1e6f7941dd8754c15320ab8b04 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 04:41:22.334232 13205 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ebdac1e6f7941dd8754c15320ab8b04 [term 2 LEADER]: Becoming Leader. State: Replica: 2ebdac1e6f7941dd8754c15320ab8b04, State: Running, Role: LEADER
I20260925 04:41:22.334614 13205 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ebdac1e6f7941dd8754c15320ab8b04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ebdac1e6f7941dd8754c15320ab8b04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34431 } }
I20260925 04:41:22.335274 13201 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ebdac1e6f7941dd8754c15320ab8b04 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:41:22.336124 13207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ebdac1e6f7941dd8754c15320ab8b04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ebdac1e6f7941dd8754c15320ab8b04. Latest consensus state: current_term: 2 leader_uuid: "2ebdac1e6f7941dd8754c15320ab8b04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ebdac1e6f7941dd8754c15320ab8b04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34431 } } }
I20260925 04:41:22.336182 13206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ebdac1e6f7941dd8754c15320ab8b04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2ebdac1e6f7941dd8754c15320ab8b04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ebdac1e6f7941dd8754c15320ab8b04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34431 } } }
I20260925 04:41:22.336302 13206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ebdac1e6f7941dd8754c15320ab8b04 [sys.catalog]: This master's current role is: LEADER
I20260925 04:41:22.336252 13207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ebdac1e6f7941dd8754c15320ab8b04 [sys.catalog]: This master's current role is: LEADER
I20260925 04:41:22.336702 13213 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:41:22.340492 13213 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:41:22.341114 13201 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 04:41:22.341277 13213 catalog_manager.cc:1259] Loaded cluster ID: 3c8f1368b8ea47f68d5de23e2f91208f
I20260925 04:41:22.341315 13213 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:41:22.347769 13213 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:41:22.349239 13213 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2ebdac1e6f7941dd8754c15320ab8b04: Loaded TSK: 0
I20260925 04:41:22.350993 13213 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 04:41:22.452672 13203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:41:22.453058 13203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:41:22.453123 13203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:41:22.459470 13203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:41:22.459654 13203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:41:22.459779 13203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 04:41:22.467456 13203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:49397/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:38785
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=39161
--tserver_master_addrs=127.0.98.62:34431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13203
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:41:22.469116 13203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:41:22.470453 13203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:41:22.482965 13239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:41:22.483171 13241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:41:22.483844 13238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:41:22.484082 13203 server_base.cc:1034] running on GCE node
I20260925 04:41:22.484593 13203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:41:22.485409 13203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:41:22.486600 13203 hybrid_clock.cc:648] HybridClock initialized: now 1790311282486583 us; error 49 us; skew 500 ppm
Sep 25 04:41:22 dist-test-slave-dc95 krb5kdc[12168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311282, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:41:22.490381 13203 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 04:41:22.491674 13203 webserver.cc:460] Webserver started at http://127.0.98.1:39161/ using document root <none> and password file <none>
I20260925 04:41:22.492488 13203 fs_manager.cc:359] Metadata directory not provided
I20260925 04:41:22.492635 13203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 04:41:22 dist-test-slave-dc95 krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311282, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 04:41:22.514184 13203 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 04:41:22.517287 13249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:41:22.519142 13203 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 04:41:22.519294 13203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-0/wal
uuid: "76ebc568de434d39b3f727e77ed7faea"
format_stamp: "Formatted at 2026-09-25 04:41:21 on dist-test-slave-dc95"
server_key: "fc87a223179576442ea0400a9b3b4d3f"
server_key_iv: "9bdc67cad8b38c3314c246e0d76e2d35"
server_key_version: "kuduclusterkey@0"
I20260925 04:41:22.520059 13203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:41:22.547138 13203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:41:22.551162 13203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:41:22.551465 13203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:41:22.552337 13203 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:41:22.553392 13203 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:41:22.553445 13203 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:41:22.553499 13203 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:41:22.553539 13203 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:41:22.563596 13203 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38785
I20260925 04:41:22.563656 13362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38785 every 8 connection(s)
I20260925 04:41:22.564970 13203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
I20260925 04:41:22.565608 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 13203
I20260925 04:41:22.566088 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:49397/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34677
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34431
--webserver_port=42389
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:49397/kms with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 04:41:22 dist-test-slave-dc95 krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311282, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:41:22.594419 13366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:22.567632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34431 (local address 127.0.98.1:44059)
0925 04:41:22.572303 (+ 4671us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:22.572380 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0925 04:41:22.573714 (+ 1334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:41:22.578081 (+ 4367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:41:22.578090 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:41:22.578410 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:41:22.579108 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:22.579132 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:22.581133 (+ 2001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:22.581136 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:22.582172 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:22.582185 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:22.583016 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:22.583020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:22.583213 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:22.583862 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:41:22.583885 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:41:22.585712 (+ 1827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:41:22.590390 (+ 4678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:41:22.590396 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:41:22.590411 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:41:22.590711 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:41:22.591181 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:41:22.591185 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:41:22.591187 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:41:22.591371 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:41:22.592289 (+ 918us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:41:22.592296 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:41:22.592676 (+ 380us) client_negotiation.cc:769] Sending connection context
0925 04:41:22.592912 (+ 236us) client_negotiation.cc:240] Negotiation successful
0925 04:41:22.593309 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4408,"thread_start_us":121,"threads_started":1}
I20260925 04:41:22.596472 13367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:22.568205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44059 (local address 127.0.98.62:34431)
0925 04:41:22.571913 (+ 3708us) server_negotiation.cc:206] Beginning negotiation
0925 04:41:22.571938 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0925 04:41:22.572576 (+ 638us) server_negotiation.cc:407] Connection header received
0925 04:41:22.576359 (+ 3783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:41:22.576381 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:41:22.576920 (+ 539us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:41:22.577324 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:41:22.579328 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:22.580932 (+ 1604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:22.582323 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:22.582830 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:22.582994 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:22.586009 (+ 3015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:41:22.586017 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:41:22.586037 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:41:22.586077 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:41:22.590203 (+ 4126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:22.590858 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:22.590865 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:22.590871 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:22.591048 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:22.591477 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:22.591480 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:22.591482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:22.592082 (+ 600us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:41:22.592224 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 04:41:22.595066 (+ 2842us) server_negotiation.cc:299] Negotiation successful
0925 04:41:22.595431 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3495,"thread_start_us":120,"threads_started":1}
I20260925 04:41:22.599371 13363 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34431
I20260925 04:41:22.599749 13363 heartbeater.cc:464] Registering TS with master...
I20260925 04:41:22.600584 13363 heartbeater.cc:511] Master 127.0.98.62:34431 requested a full tablet report, sending...
W20260925 04:41:22.603097 13363 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34431 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 04:41:22.603371 13363 heartbeater.cc:464] Registering TS with master...
I20260925 04:41:22.603701 13363 heartbeater.cc:511] Master 127.0.98.62:34431 requested a full tablet report, sending...
I20260925 04:41:22.604861 13363 heartbeater.cc:464] Registering TS with master...
I20260925 04:41:22.605152 13363 heartbeater.cc:511] Master 127.0.98.62:34431 requested a full tablet report, sending...
W20260925 04:41:22.606065 13363 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 04:41:22.738134 13365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:41:22.738436 13365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:41:22.738497 13365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:41:22.743494 13365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:41:22.743635 13365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:41:22.743733 13365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 04:41:22.752267 13365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:49397/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34677
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=42389
--tserver_master_addrs=127.0.98.62:34431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:41:22.753984 13365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:41:22.755482 13365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:41:22.769078 13373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:41:22.769104 13375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:41:22.769292 13372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:41:22.773200 13365 server_base.cc:1034] running on GCE node
I20260925 04:41:22.773784 13365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:41:22.774683 13365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:41:22.782155 13365 hybrid_clock.cc:648] HybridClock initialized: now 1790311282782093 us; error 68 us; skew 500 ppm
Sep 25 04:41:22 dist-test-slave-dc95 krb5kdc[12168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311282, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:41:22.786154 13365 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 04:41:22.787487 13365 webserver.cc:460] Webserver started at http://127.0.98.2:42389/ using document root <none> and password file <none>
I20260925 04:41:22.788317 13365 fs_manager.cc:359] Metadata directory not provided
I20260925 04:41:22.788435 13365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 04:41:22 dist-test-slave-dc95 krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311282, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 04:41:22.818198 13365 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.003s sys 0.000s
I20260925 04:41:22.822050 13383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:41:22.823943 13365 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260925 04:41:22.824136 13365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-1/wal
uuid: "8de887c7a4c1454384a914454b9e6457"
format_stamp: "Formatted at 2026-09-25 04:41:21 on dist-test-slave-dc95"
server_key: "c0bc54f554ed32f4d7c931f86594447b"
server_key_iv: "d8c319efe95b4359c43d2c59d7295a28"
server_key_version: "kuduclusterkey@0"
I20260925 04:41:22.824944 13365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:41:22.860906 13365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:41:22.865597 13365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:41:22.865897 13365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:41:22.866617 13365 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:41:22.867821 13365 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:41:22.867877 13365 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:41:22.867933 13365 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:41:22.867965 13365 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:41:22.879422 13365 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34677
I20260925 04:41:22.879547 13497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34677 every 8 connection(s)
I20260925 04:41:22.880959 13365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
I20260925 04:41:22.887265 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 13365
I20260925 04:41:22.887722 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:49397/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33591
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34431
--webserver_port=39377
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:49397/kms with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 04:41:22 dist-test-slave-dc95 krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311282, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:41:22.898077 13367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:22.883812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39757 (local address 127.0.98.62:34431)
0925 04:41:22.883967 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 04:41:22.883972 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:41:22.884842 (+ 870us) server_negotiation.cc:407] Connection header received
0925 04:41:22.886106 (+ 1264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:41:22.886109 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:41:22.886168 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:41:22.886272 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:41:22.887920 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:22.888743 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:22.890549 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:22.890837 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:22.890949 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:22.893997 (+ 3048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:41:22.894001 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:41:22.894004 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:41:22.894038 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:41:22.895773 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:22.896476 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:22.896480 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:22.896482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:22.896569 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:22.896927 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:22.896931 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:22.896933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:22.897128 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:41:22.897216 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 04:41:22.897773 (+ 557us) server_negotiation.cc:299] Negotiation successful
0925 04:41:22.897908 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 04:41:22.898967 13500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:22.884125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34431 (local address 127.0.98.2:39757)
0925 04:41:22.884637 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:22.884701 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0925 04:41:22.885827 (+ 1126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:41:22.886512 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:41:22.886522 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:41:22.886905 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:41:22.887707 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:22.887731 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:22.888928 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:22.888931 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:22.890398 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:22.890411 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:22.891156 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:22.891160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:22.891386 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:22.892112 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:41:22.892134 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:41:22.893768 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:41:22.895917 (+ 2149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:41:22.895926 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:41:22.895947 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:41:22.896337 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:41:22.896655 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:41:22.896658 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:41:22.896660 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:41:22.896823 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:41:22.897238 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:41:22.897245 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:41:22.897526 (+ 281us) client_negotiation.cc:769] Sending connection context
0925 04:41:22.897703 (+ 177us) client_negotiation.cc:240] Negotiation successful
0925 04:41:22.898042 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":117,"threads_started":1}
I20260925 04:41:22.900287 13498 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34431
I20260925 04:41:22.900571 13498 heartbeater.cc:464] Registering TS with master...
I20260925 04:41:22.901214 13498 heartbeater.cc:511] Master 127.0.98.62:34431 requested a full tablet report, sending...
W20260925 04:41:22.903187 13498 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34431 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 04:41:22.903311 13498 heartbeater.cc:464] Registering TS with master...
I20260925 04:41:22.903483 13498 heartbeater.cc:511] Master 127.0.98.62:34431 requested a full tablet report, sending...
I20260925 04:41:22.904325 13498 heartbeater.cc:464] Registering TS with master...
I20260925 04:41:22.904516 13498 heartbeater.cc:511] Master 127.0.98.62:34431 requested a full tablet report, sending...
W20260925 04:41:22.905246 13498 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 04:41:23.028035 13501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:41:23.028549 13501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:41:23.028666 13501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:41:23.034205 13501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:41:23.034286 13501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:41:23.034349 13501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 04:41:23.041131 13501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:49397/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33591
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=39377
--tserver_master_addrs=127.0.98.62:34431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:41:23.042685 13501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:41:23.044157 13501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:41:23.054792 13506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:41:23.055195 13509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:41:23.055322 13507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:41:23.055574 13501 server_base.cc:1034] running on GCE node
I20260925 04:41:23.056119 13501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:41:23.056908 13501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:41:23.058070 13501 hybrid_clock.cc:648] HybridClock initialized: now 1790311283058050 us; error 32 us; skew 500 ppm
Sep 25 04:41:23 dist-test-slave-dc95 krb5kdc[12168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311283, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:41:23.062143 13501 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 04:41:23.063498 13501 webserver.cc:460] Webserver started at http://127.0.98.3:39377/ using document root <none> and password file <none>
I20260925 04:41:23.064360 13501 fs_manager.cc:359] Metadata directory not provided
I20260925 04:41:23.064477 13501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 04:41:23 dist-test-slave-dc95 krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311283, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 04:41:23.087527 13501 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260925 04:41:23.090165 13517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:41:23.091504 13501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 04:41:23.091667 13501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-2/wal
uuid: "d622c3c03bcf4214904aa23c1ebd7ff9"
format_stamp: "Formatted at 2026-09-25 04:41:21 on dist-test-slave-dc95"
server_key: "d6cb89a9e0fbb66d5c7ff42c4d5a98b2"
server_key_iv: "d425b4a83c5167d893a94a961e307f68"
server_key_version: "kuduclusterkey@0"
I20260925 04:41:23.092248 13501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:41:23.103744 13501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:41:23.107283 13501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:41:23.107556 13501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:41:23.108318 13501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:41:23.109319 13501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:41:23.109368 13501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:41:23.109418 13501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:41:23.109448 13501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:41:23.120191 13501 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33591
I20260925 04:41:23.121287 13501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
I20260925 04:41:23.122607 13637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33591 every 8 connection(s)
I20260925 04:41:23.126060 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 13501
Sep 25 04:41:23 dist-test-slave-dc95 krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311283, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:41:23.137068 13642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:23.126799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56350 (local address 127.0.98.62:34431)
0925 04:41:23.127658 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0925 04:41:23.127667 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:41:23.127680 (+ 13us) server_negotiation.cc:407] Connection header received
0925 04:41:23.127715 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:41:23.127718 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:41:23.127767 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:41:23.127841 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:41:23.128814 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:23.129664 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:23.131674 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:23.131931 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:23.132032 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:23.133728 (+ 1696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:41:23.133730 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:41:23.133732 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:41:23.133773 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:41:23.135257 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:23.135784 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:23.135786 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:23.135787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:23.135857 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:23.136171 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:23.136175 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:23.136176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:23.136362 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:41:23.136548 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0925 04:41:23.136757 (+ 209us) server_negotiation.cc:299] Negotiation successful
0925 04:41:23.136876 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":120,"threads_started":1}
I20260925 04:41:23.137388 13367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:23.123279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35927 (local address 127.0.98.62:34431)
0925 04:41:23.123428 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 04:41:23.123433 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:41:23.124379 (+ 946us) server_negotiation.cc:407] Connection header received
0925 04:41:23.125377 (+ 998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:41:23.125381 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:41:23.125437 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:41:23.125504 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:41:23.126822 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:23.127549 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:23.129174 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:23.129397 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:23.129498 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:23.132636 (+ 3138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:41:23.132641 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:41:23.132644 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:41:23.132672 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:41:23.134406 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:23.135233 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:23.135237 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:23.135239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:23.135398 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:23.135901 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:23.135904 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:23.135906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:23.136159 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:41:23.136270 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 04:41:23.137103 (+ 833us) server_negotiation.cc:299] Negotiation successful
0925 04:41:23.137236 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 04:41:23.139801 13640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:23.123743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34431 (local address 127.0.98.3:35927)
0925 04:41:23.124200 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:23.124252 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 04:41:23.125154 (+ 902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:41:23.125701 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:41:23.125708 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:41:23.125983 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:41:23.126648 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:23.126672 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:23.127752 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:23.127755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:23.129050 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:23.129060 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:23.129574 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:23.129578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:23.129843 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:23.130482 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:41:23.130503 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:41:23.132443 (+ 1940us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:41:23.134706 (+ 2263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:41:23.134713 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:41:23.134734 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:41:23.135089 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:41:23.135516 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:41:23.135519 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:41:23.135521 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:41:23.135787 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:41:23.136340 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:41:23.136347 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:41:23.136815 (+ 468us) client_negotiation.cc:769] Sending connection context
0925 04:41:23.137599 (+ 784us) client_negotiation.cc:240] Negotiation successful
0925 04:41:23.138625 (+ 1026us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":98,"threads_started":1}
I20260925 04:41:23.140951 13638 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34431
I20260925 04:41:23.141213 13638 heartbeater.cc:464] Registering TS with master...
I20260925 04:41:23.141710 13638 heartbeater.cc:511] Master 127.0.98.62:34431 requested a full tablet report, sending...
I20260925 04:41:23.143036 13163 ts_manager.cc:194] Registered new tserver with Master: d622c3c03bcf4214904aa23c1ebd7ff9 (127.0.98.3:33591)
I20260925 04:41:23.144214 13163 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35927
Sep 25 04:41:23 dist-test-slave-dc95 krb5kdc[12168](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790311283, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:41:23.606369 13363 heartbeater.cc:464] Registering TS with master...
I20260925 04:41:23.606683 13363 heartbeater.cc:511] Master 127.0.98.62:34431 requested a full tablet report, sending...
I20260925 04:41:23.607645 13163 ts_manager.cc:194] Registered new tserver with Master: 76ebc568de434d39b3f727e77ed7faea (127.0.98.1:38785)
I20260925 04:41:23.608560 13163 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44059
Sep 25 04:41:23 dist-test-slave-dc95 krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311283, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 04:41:23.905507 13498 heartbeater.cc:464] Registering TS with master...
I20260925 04:41:23.905822 13498 heartbeater.cc:511] Master 127.0.98.62:34431 requested a full tablet report, sending...
I20260925 04:41:23.906697 13163 ts_manager.cc:194] Registered new tserver with Master: 8de887c7a4c1454384a914454b9e6457 (127.0.98.2:34677)
I20260925 04:41:23.907523 13163 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39757
I20260925 04:41:23.913228 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:41:23.929997 13652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:23.915790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56362 (local address 127.0.98.62:34431)
0925 04:41:23.919821 (+ 4031us) server_negotiation.cc:206] Beginning negotiation
0925 04:41:23.919826 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:41:23.920066 (+ 240us) server_negotiation.cc:407] Connection header received
0925 04:41:23.920124 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:41:23.920127 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:41:23.920183 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:41:23.920279 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:41:23.921265 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:23.922168 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:23.923495 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:23.923911 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:23.924014 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:23.926049 (+ 2035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:41:23.926053 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:41:23.926056 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:41:23.926092 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:41:23.927872 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:23.928394 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:23.928397 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:23.928399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:23.928498 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:23.928784 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:23.928787 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:23.928789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:23.929027 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:41:23.929148 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 04:41:23.929673 (+ 525us) server_negotiation.cc:299] Negotiation successful
0925 04:41:23.929811 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3913,"thread_start_us":99,"threads_started":1}
I20260925 04:41:23.942354 13163 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56362:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:41:23.947523 13163 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:41:23.988722 13658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:23.971593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38785 (local address 127.0.0.1:43898)
0925 04:41:23.979975 (+ 8382us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:23.980053 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0925 04:41:23.980212 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:41:23.981186 (+ 974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:41:23.981193 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:41:23.981216 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:41:23.981379 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:23.981386 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:23.984208 (+ 2822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:23.984212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:23.985618 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:23.985631 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:23.987068 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:23.987072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:23.987201 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:23.987247 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 04:41:23.987373 (+ 126us) client_negotiation.cc:240] Negotiation successful
0925 04:41:23.987518 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8137,"thread_start_us":124,"threads_started":1}
I20260925 04:41:23.991389 13659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:23.971765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43898 (local address 127.0.98.1:38785)
0925 04:41:23.979877 (+ 8112us) server_negotiation.cc:206] Beginning negotiation
0925 04:41:23.979885 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:41:23.980431 (+ 546us) server_negotiation.cc:407] Connection header received
0925 04:41:23.980519 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:41:23.980525 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:41:23.980781 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:41:23.980925 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:41:23.981880 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:23.984063 (+ 2183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:23.985782 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:23.986721 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:23.986835 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:23.986998 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 04:41:23.991095 (+ 4097us) server_negotiation.cc:299] Negotiation successful
0925 04:41:23.991229 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7980,"thread_start_us":104,"threads_started":1}
I20260925 04:41:23.994864 13662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:23.972216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33591 (local address 127.0.0.1:53824)
0925 04:41:23.981173 (+ 8957us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:23.981185 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:41:23.981265 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:41:23.989331 (+ 8066us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:41:23.989334 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:41:23.989370 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:41:23.989480 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:23.989485 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:23.990993 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:23.990996 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:23.992194 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:23.992205 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:23.994451 (+ 2246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:23.994455 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:23.994544 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:23.994571 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 04:41:23.994630 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 04:41:23.994709 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8894,"mutex_wait_us":418,"thread_start_us":95,"threads_started":1}
I20260925 04:41:23.994904 13297 tablet_service.cc:1467] Processing CreateTablet for tablet e0420148047a43bebfbf096da8573675 (DEFAULT_TABLE table=test-table [id=9c2fa3e875b9487dbad330267e7eb5f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:41:23.995468 13661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:23.972426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53824 (local address 127.0.98.3:33591)
0925 04:41:23.980067 (+ 7641us) server_negotiation.cc:206] Beginning negotiation
0925 04:41:23.980072 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:41:23.981368 (+ 1296us) server_negotiation.cc:407] Connection header received
0925 04:41:23.981449 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:41:23.981453 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:41:23.981621 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:41:23.981735 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:41:23.989646 (+ 7911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:23.990844 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:23.992343 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:23.993183 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:23.993273 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:23.994393 (+ 1120us) server_negotiation.cc:1035] Waiting for connection context
0925 04:41:23.995196 (+ 803us) server_negotiation.cc:299] Negotiation successful
0925 04:41:23.995310 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7542,"thread_start_us":78,"threads_started":1}
I20260925 04:41:23.995739 13660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:23.972213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34677 (local address 127.0.0.1:51746)
0925 04:41:23.980214 (+ 8001us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:23.980227 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:41:23.980316 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:41:23.991677 (+ 11361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:41:23.991681 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:41:23.991709 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:41:23.991813 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:23.991820 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:23.993247 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:23.993251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:23.994217 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:23.994226 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:23.995353 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:23.995357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:23.995446 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:23.995471 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 04:41:23.995521 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 04:41:23.995587 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7934,"thread_start_us":122,"threads_started":1}
I20260925 04:41:23.996392 13663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:23.972771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51746 (local address 127.0.98.2:34677)
0925 04:41:23.980525 (+ 7754us) server_negotiation.cc:206] Beginning negotiation
0925 04:41:23.980531 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:41:23.980554 (+ 23us) server_negotiation.cc:407] Connection header received
0925 04:41:23.980629 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:41:23.980634 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:41:23.980845 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:41:23.980991 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:41:23.991974 (+ 10983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:23.993103 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:23.994378 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:23.995233 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:23.995914 (+ 681us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:23.996058 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 04:41:23.996108 (+ 50us) server_negotiation.cc:299] Negotiation successful
0925 04:41:23.996197 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7646,"thread_start_us":88,"threads_started":1}
I20260925 04:41:23.999164 13413 tablet_service.cc:1467] Processing CreateTablet for tablet e0420148047a43bebfbf096da8573675 (DEFAULT_TABLE table=test-table [id=9c2fa3e875b9487dbad330267e7eb5f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:41:23.999169 13572 tablet_service.cc:1467] Processing CreateTablet for tablet e0420148047a43bebfbf096da8573675 (DEFAULT_TABLE table=test-table [id=9c2fa3e875b9487dbad330267e7eb5f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:41:24.000332 13413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0420148047a43bebfbf096da8573675. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:41:24.000376 13572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0420148047a43bebfbf096da8573675. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:41:24.004302 13297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0420148047a43bebfbf096da8573675. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:41:24.017339 13665 tablet_bootstrap.cc:492] T e0420148047a43bebfbf096da8573675 P 8de887c7a4c1454384a914454b9e6457: Bootstrap starting.
I20260925 04:41:24.019909 13665 tablet_bootstrap.cc:654] T e0420148047a43bebfbf096da8573675 P 8de887c7a4c1454384a914454b9e6457: Neither blocks nor log segments found. Creating new log.
I20260925 04:41:24.020284 13666 tablet_bootstrap.cc:492] T e0420148047a43bebfbf096da8573675 P 76ebc568de434d39b3f727e77ed7faea: Bootstrap starting.
I20260925 04:41:24.020845 13665 log.cc:824] T e0420148047a43bebfbf096da8573675 P 8de887c7a4c1454384a914454b9e6457: Log is configured to *not* fsync() on all Append() calls
I20260925 04:41:24.021745 13664 tablet_bootstrap.cc:492] T e0420148047a43bebfbf096da8573675 P d622c3c03bcf4214904aa23c1ebd7ff9: Bootstrap starting.
I20260925 04:41:24.023543 13666 tablet_bootstrap.cc:654] T e0420148047a43bebfbf096da8573675 P 76ebc568de434d39b3f727e77ed7faea: Neither blocks nor log segments found. Creating new log.
I20260925 04:41:24.024542 13666 log.cc:824] T e0420148047a43bebfbf096da8573675 P 76ebc568de434d39b3f727e77ed7faea: Log is configured to *not* fsync() on all Append() calls
I20260925 04:41:24.024726 13664 tablet_bootstrap.cc:654] T e0420148047a43bebfbf096da8573675 P d622c3c03bcf4214904aa23c1ebd7ff9: Neither blocks nor log segments found. Creating new log.
I20260925 04:41:24.025573 13664 log.cc:824] T e0420148047a43bebfbf096da8573675 P d622c3c03bcf4214904aa23c1ebd7ff9: Log is configured to *not* fsync() on all Append() calls
I20260925 04:41:24.026950 13666 tablet_bootstrap.cc:492] T e0420148047a43bebfbf096da8573675 P 76ebc568de434d39b3f727e77ed7faea: No bootstrap required, opened a new log
I20260925 04:41:24.027140 13666 ts_tablet_manager.cc:1397] T e0420148047a43bebfbf096da8573675 P 76ebc568de434d39b3f727e77ed7faea: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 04:41:24.027855 13665 tablet_bootstrap.cc:492] T e0420148047a43bebfbf096da8573675 P 8de887c7a4c1454384a914454b9e6457: No bootstrap required, opened a new log
I20260925 04:41:24.028051 13665 ts_tablet_manager.cc:1397] T e0420148047a43bebfbf096da8573675 P 8de887c7a4c1454384a914454b9e6457: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20260925 04:41:24.032023 13666 raft_consensus.cc:348] T e0420148047a43bebfbf096da8573675 P 76ebc568de434d39b3f727e77ed7faea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76ebc568de434d39b3f727e77ed7faea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38785 } } peers { permanent_uuid: "8de887c7a4c1454384a914454b9e6457" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34677 } } peers { permanent_uuid: "d622c3c03bcf4214904aa23c1ebd7ff9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33591 } }
I20260925 04:41:24.032033 13665 raft_consensus.cc:348] T e0420148047a43bebfbf096da8573675 P 8de887c7a4c1454384a914454b9e6457 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76ebc568de434d39b3f727e77ed7faea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38785 } } peers { permanent_uuid: "8de887c7a4c1454384a914454b9e6457" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34677 } } peers { permanent_uuid: "d622c3c03bcf4214904aa23c1ebd7ff9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33591 } }
I20260925 04:41:24.032305 13665 raft_consensus.cc:374] T e0420148047a43bebfbf096da8573675 P 8de887c7a4c1454384a914454b9e6457 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:41:24.032387 13665 raft_consensus.cc:729] T e0420148047a43bebfbf096da8573675 P 8de887c7a4c1454384a914454b9e6457 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8de887c7a4c1454384a914454b9e6457, State: Initialized, Role: FOLLOWER
I20260925 04:41:24.032433 13666 raft_consensus.cc:374] T e0420148047a43bebfbf096da8573675 P 76ebc568de434d39b3f727e77ed7faea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:41:24.032562 13666 raft_consensus.cc:729] T e0420148047a43bebfbf096da8573675 P 76ebc568de434d39b3f727e77ed7faea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76ebc568de434d39b3f727e77ed7faea, State: Initialized, Role: FOLLOWER
I20260925 04:41:24.032910 13665 consensus_queue.cc:260] T e0420148047a43bebfbf096da8573675 P 8de887c7a4c1454384a914454b9e6457 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76ebc568de434d39b3f727e77ed7faea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38785 } } peers { permanent_uuid: "8de887c7a4c1454384a914454b9e6457" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34677 } } peers { permanent_uuid: "d622c3c03bcf4214904aa23c1ebd7ff9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33591 } }
I20260925 04:41:24.033183 13666 consensus_queue.cc:260] T e0420148047a43bebfbf096da8573675 P 76ebc568de434d39b3f727e77ed7faea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76ebc568de434d39b3f727e77ed7faea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38785 } } peers { permanent_uuid: "8de887c7a4c1454384a914454b9e6457" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34677 } } peers { permanent_uuid: "d622c3c03bcf4214904aa23c1ebd7ff9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33591 } }
I20260925 04:41:24.034025 13665 ts_tablet_manager.cc:1428] T e0420148047a43bebfbf096da8573675 P 8de887c7a4c1454384a914454b9e6457: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260925 04:41:24.034219 13666 ts_tablet_manager.cc:1428] T e0420148047a43bebfbf096da8573675 P 76ebc568de434d39b3f727e77ed7faea: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260925 04:41:24.034441 13498 heartbeater.cc:503] Master 127.0.98.62:34431 was elected leader, sending a full tablet report...
I20260925 04:41:24.036515 13664 tablet_bootstrap.cc:492] T e0420148047a43bebfbf096da8573675 P d622c3c03bcf4214904aa23c1ebd7ff9: No bootstrap required, opened a new log
I20260925 04:41:24.036770 13664 ts_tablet_manager.cc:1397] T e0420148047a43bebfbf096da8573675 P d622c3c03bcf4214904aa23c1ebd7ff9: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.000s
I20260925 04:41:24.039196 13363 heartbeater.cc:503] Master 127.0.98.62:34431 was elected leader, sending a full tablet report...
I20260925 04:41:24.042014 13664 raft_consensus.cc:348] T e0420148047a43bebfbf096da8573675 P d622c3c03bcf4214904aa23c1ebd7ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76ebc568de434d39b3f727e77ed7faea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38785 } } peers { permanent_uuid: "8de887c7a4c1454384a914454b9e6457" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34677 } } peers { permanent_uuid: "d622c3c03bcf4214904aa23c1ebd7ff9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33591 } }
I20260925 04:41:24.042336 13664 raft_consensus.cc:374] T e0420148047a43bebfbf096da8573675 P d622c3c03bcf4214904aa23c1ebd7ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:41:24.042402 13664 raft_consensus.cc:729] T e0420148047a43bebfbf096da8573675 P d622c3c03bcf4214904aa23c1ebd7ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d622c3c03bcf4214904aa23c1ebd7ff9, State: Initialized, Role: FOLLOWER
I20260925 04:41:24.042987 13664 consensus_queue.cc:260] T e0420148047a43bebfbf096da8573675 P d622c3c03bcf4214904aa23c1ebd7ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76ebc568de434d39b3f727e77ed7faea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38785 } } peers { permanent_uuid: "8de887c7a4c1454384a914454b9e6457" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34677 } } peers { permanent_uuid: "d622c3c03bcf4214904aa23c1ebd7ff9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33591 } }
I20260925 04:41:24.044152 13664 ts_tablet_manager.cc:1428] T e0420148047a43bebfbf096da8573675 P d622c3c03bcf4214904aa23c1ebd7ff9: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260925 04:41:24.044535 13638 heartbeater.cc:503] Master 127.0.98.62:34431 was elected leader, sending a full tablet report...
W20260925 04:41:24.067164 13364 tablet.cc:2369] T e0420148047a43bebfbf096da8573675 P 76ebc568de434d39b3f727e77ed7faea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:41:24.114264 13670 raft_consensus.cc:482] T e0420148047a43bebfbf096da8573675 P 8de887c7a4c1454384a914454b9e6457 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:41:24.114626 13670 raft_consensus.cc:504] T e0420148047a43bebfbf096da8573675 P 8de887c7a4c1454384a914454b9e6457 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76ebc568de434d39b3f727e77ed7faea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38785 } } peers { permanent_uuid: "8de887c7a4c1454384a914454b9e6457" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34677 } } peers { permanent_uuid: "d622c3c03bcf4214904aa23c1ebd7ff9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33591 } }
I20260925 04:41:24.116257 13670 leader_election.cc:288] T e0420148047a43bebfbf096da8573675 P 8de887c7a4c1454384a914454b9e6457 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76ebc568de434d39b3f727e77ed7faea (127.0.98.1:38785), d622c3c03bcf4214904aa23c1ebd7ff9 (127.0.98.3:33591)
I20260925 04:41:24.121590 13659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:24.116965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33103 (local address 127.0.98.1:38785)
0925 04:41:24.117110 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 04:41:24.117114 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:41:24.117252 (+ 138us) server_negotiation.cc:407] Connection header received
0925 04:41:24.117452 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:41:24.117456 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:41:24.117504 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:41:24.117549 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:41:24.117961 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:24.118689 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:24.120302 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:24.120904 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:24.120978 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:24.121044 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 04:41:24.121386 (+ 342us) server_negotiation.cc:299] Negotiation successful
0925 04:41:24.121466 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 04:41:24.122274 13673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:24.116531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38785 (local address 127.0.98.2:33103)
0925 04:41:24.117153 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:24.117168 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:41:24.117334 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:41:24.117679 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:41:24.117682 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:41:24.117699 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:41:24.117851 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:24.117857 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:24.118839 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:24.118842 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:24.120164 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:24.120174 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:24.121159 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:24.121163 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:24.121259 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:24.121281 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 04:41:24.122033 (+ 752us) client_negotiation.cc:240] Negotiation successful
0925 04:41:24.122119 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":136,"threads_started":1}
I20260925 04:41:24.123083 13317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0420148047a43bebfbf096da8573675" candidate_uuid: "8de887c7a4c1454384a914454b9e6457" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76ebc568de434d39b3f727e77ed7faea" is_pre_election: true
I20260925 04:41:24.123399 13317 raft_consensus.cc:2454] T e0420148047a43bebfbf096da8573675 P 76ebc568de434d39b3f727e77ed7faea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de887c7a4c1454384a914454b9e6457 in term 0.
W20260925 04:41:24.123793 13639 tablet.cc:2369] T e0420148047a43bebfbf096da8573675 P d622c3c03bcf4214904aa23c1ebd7ff9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:41:24.124598 13385 leader_election.cc:302] T e0420148047a43bebfbf096da8573675 P 8de887c7a4c1454384a914454b9e6457 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76ebc568de434d39b3f727e77ed7faea, 8de887c7a4c1454384a914454b9e6457; no voters:
I20260925 04:41:24.124907 13670 raft_consensus.cc:2781] T e0420148047a43bebfbf096da8573675 P 8de887c7a4c1454384a914454b9e6457 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:41:24.125007 13670 raft_consensus.cc:482] T e0420148047a43bebfbf096da8573675 P 8de887c7a4c1454384a914454b9e6457 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:41:24.125056 13670 raft_consensus.cc:3037] T e0420148047a43bebfbf096da8573675 P 8de887c7a4c1454384a914454b9e6457 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:41:24.127177 13670 raft_consensus.cc:504] T e0420148047a43bebfbf096da8573675 P 8de887c7a4c1454384a914454b9e6457 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76ebc568de434d39b3f727e77ed7faea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38785 } } peers { permanent_uuid: "8de887c7a4c1454384a914454b9e6457" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34677 } } peers { permanent_uuid: "d622c3c03bcf4214904aa23c1ebd7ff9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33591 } }
I20260925 04:41:24.128034 13317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0420148047a43bebfbf096da8573675" candidate_uuid: "8de887c7a4c1454384a914454b9e6457" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76ebc568de434d39b3f727e77ed7faea"
I20260925 04:41:24.128209 13317 raft_consensus.cc:3037] T e0420148047a43bebfbf096da8573675 P 76ebc568de434d39b3f727e77ed7faea [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:41:24.128401 13661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:24.117224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59945 (local address 127.0.98.3:33591)
0925 04:41:24.117347 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 04:41:24.117351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:41:24.117750 (+ 399us) server_negotiation.cc:407] Connection header received
0925 04:41:24.117948 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:41:24.117952 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:41:24.118023 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:41:24.118072 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:41:24.118473 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:24.119235 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:24.127066 (+ 7831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:24.127687 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:24.127783 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:24.127849 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 04:41:24.128186 (+ 337us) server_negotiation.cc:299] Negotiation successful
0925 04:41:24.128282 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 04:41:24.128760 13674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:24.116676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33591 (local address 127.0.98.2:59945)
0925 04:41:24.117636 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:24.117651 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:41:24.117829 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:41:24.118212 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:41:24.118215 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:41:24.118231 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:41:24.118350 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:24.118356 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:24.119409 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:24.119412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:24.126905 (+ 7493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:24.126919 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:24.127948 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:24.127951 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:24.128053 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:24.128074 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 04:41:24.128537 (+ 463us) client_negotiation.cc:240] Negotiation successful
0925 04:41:24.128618 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":887,"thread_start_us":102,"threads_started":1}
I20260925 04:41:24.129436 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0420148047a43bebfbf096da8573675" candidate_uuid: "8de887c7a4c1454384a914454b9e6457" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d622c3c03bcf4214904aa23c1ebd7ff9" is_pre_election: true
I20260925 04:41:24.129531 13317 raft_consensus.cc:2454] T e0420148047a43bebfbf096da8573675 P 76ebc568de434d39b3f727e77ed7faea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de887c7a4c1454384a914454b9e6457 in term 1.
I20260925 04:41:24.129734 13588 raft_consensus.cc:2454] T e0420148047a43bebfbf096da8573675 P d622c3c03bcf4214904aa23c1ebd7ff9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de887c7a4c1454384a914454b9e6457 in term 0.
I20260925 04:41:24.129889 13385 leader_election.cc:302] T e0420148047a43bebfbf096da8573675 P 8de887c7a4c1454384a914454b9e6457 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76ebc568de434d39b3f727e77ed7faea, 8de887c7a4c1454384a914454b9e6457; no voters:
I20260925 04:41:24.130576 13675 raft_consensus.cc:2781] T e0420148047a43bebfbf096da8573675 P 8de887c7a4c1454384a914454b9e6457 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:41:24.130869 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0420148047a43bebfbf096da8573675" candidate_uuid: "8de887c7a4c1454384a914454b9e6457" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d622c3c03bcf4214904aa23c1ebd7ff9"
I20260925 04:41:24.130908 13675 raft_consensus.cc:686] T e0420148047a43bebfbf096da8573675 P 8de887c7a4c1454384a914454b9e6457 [term 1 LEADER]: Becoming Leader. State: Replica: 8de887c7a4c1454384a914454b9e6457, State: Running, Role: LEADER
I20260925 04:41:24.130980 13588 raft_consensus.cc:3037] T e0420148047a43bebfbf096da8573675 P d622c3c03bcf4214904aa23c1ebd7ff9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:41:24.131330 13675 consensus_queue.cc:237] T e0420148047a43bebfbf096da8573675 P 8de887c7a4c1454384a914454b9e6457 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76ebc568de434d39b3f727e77ed7faea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38785 } } peers { permanent_uuid: "8de887c7a4c1454384a914454b9e6457" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34677 } } peers { permanent_uuid: "d622c3c03bcf4214904aa23c1ebd7ff9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33591 } }
I20260925 04:41:24.132207 13588 raft_consensus.cc:2454] T e0420148047a43bebfbf096da8573675 P d622c3c03bcf4214904aa23c1ebd7ff9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de887c7a4c1454384a914454b9e6457 in term 1.
I20260925 04:41:24.132632 13670 leader_election.cc:288] T e0420148047a43bebfbf096da8573675 P 8de887c7a4c1454384a914454b9e6457 [CANDIDATE]: Term 1 election: Requested vote from peers 76ebc568de434d39b3f727e77ed7faea (127.0.98.1:38785), d622c3c03bcf4214904aa23c1ebd7ff9 (127.0.98.3:33591)
W20260925 04:41:24.132959 13499 tablet.cc:2369] T e0420148047a43bebfbf096da8573675 P 8de887c7a4c1454384a914454b9e6457: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:41:24.135015 13163 catalog_manager.cc:5495] T e0420148047a43bebfbf096da8573675 P 8de887c7a4c1454384a914454b9e6457 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8de887c7a4c1454384a914454b9e6457 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "8de887c7a4c1454384a914454b9e6457" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76ebc568de434d39b3f727e77ed7faea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de887c7a4c1454384a914454b9e6457" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d622c3c03bcf4214904aa23c1ebd7ff9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33591 } health_report { overall_health: UNKNOWN } } }
I20260925 04:41:24.173900 13663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:24.168508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51754 (local address 127.0.98.2:34677)
0925 04:41:24.168622 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 04:41:24.168626 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:41:24.168681 (+ 55us) server_negotiation.cc:407] Connection header received
0925 04:41:24.168886 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:41:24.168889 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:41:24.168953 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:41:24.169052 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:41:24.169434 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:24.170255 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:24.171493 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:24.171868 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:24.171992 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:24.172462 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:41:24.173059 (+ 597us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:41:24.173180 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 04:41:24.173635 (+ 455us) server_negotiation.cc:299] Negotiation successful
0925 04:41:24.173706 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 04:41:24.181803 13317 raft_consensus.cc:1261] T e0420148047a43bebfbf096da8573675 P 76ebc568de434d39b3f727e77ed7faea [term 1 FOLLOWER]: Refusing update from remote peer 8de887c7a4c1454384a914454b9e6457: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:41:24.181913 13588 raft_consensus.cc:1261] T e0420148047a43bebfbf096da8573675 P d622c3c03bcf4214904aa23c1ebd7ff9 [term 1 FOLLOWER]: Refusing update from remote peer 8de887c7a4c1454384a914454b9e6457: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:41:24.182757 13676 consensus_queue.cc:1035] T e0420148047a43bebfbf096da8573675 P 8de887c7a4c1454384a914454b9e6457 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76ebc568de434d39b3f727e77ed7faea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:41:24.183149 13670 consensus_queue.cc:1035] T e0420148047a43bebfbf096da8573675 P 8de887c7a4c1454384a914454b9e6457 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d622c3c03bcf4214904aa23c1ebd7ff9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Sep 25 04:41:24 dist-test-slave-dc95 krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311283, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 04:41:24.196579 13680 mvcc.cc:205] Tried to move back new op lower bound from 7333115019999498240 to 7333115019805511680. Current Snapshot: MvccSnapshot[applied={T|T < 7333115019999498240}]
I20260925 04:41:24.201463 13681 mvcc.cc:205] Tried to move back new op lower bound from 7333115019999498240 to 7333115019805511680. Current Snapshot: MvccSnapshot[applied={T|T < 7333115019999498240}]
I20260925 04:41:24.211862 13165 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56362:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:41:24.212061 13165 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56362:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:41:24.215209 13165 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2ebdac1e6f7941dd8754c15320ab8b04: Sending DeleteTablet for 3 replicas of tablet e0420148047a43bebfbf096da8573675
I20260925 04:41:24.216153 13413 tablet_service.cc:1514] Processing DeleteTablet for tablet e0420148047a43bebfbf096da8573675 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:41:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51746
I20260925 04:41:24.217056 13297 tablet_service.cc:1514] Processing DeleteTablet for tablet e0420148047a43bebfbf096da8573675 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:41:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43898
I20260925 04:41:24.218509 13572 tablet_service.cc:1514] Processing DeleteTablet for tablet e0420148047a43bebfbf096da8573675 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:41:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53824
I20260925 04:41:24.219667 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 13203
I20260925 04:41:24.224635 13691 tablet_replica.cc:330] T e0420148047a43bebfbf096da8573675 P d622c3c03bcf4214904aa23c1ebd7ff9: stopping tablet replica
I20260925 04:41:24.225101 13691 raft_consensus.cc:2229] T e0420148047a43bebfbf096da8573675 P d622c3c03bcf4214904aa23c1ebd7ff9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:41:24.225474 13689 tablet_replica.cc:330] T e0420148047a43bebfbf096da8573675 P 8de887c7a4c1454384a914454b9e6457: stopping tablet replica
I20260925 04:41:24.229527 13691 raft_consensus.cc:2258] T e0420148047a43bebfbf096da8573675 P d622c3c03bcf4214904aa23c1ebd7ff9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:41:24.230154 13689 raft_consensus.cc:2229] T e0420148047a43bebfbf096da8573675 P 8de887c7a4c1454384a914454b9e6457 [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:41:24.230805 13689 raft_consensus.cc:2258] T e0420148047a43bebfbf096da8573675 P 8de887c7a4c1454384a914454b9e6457 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:41:24.232887 13691 ts_tablet_manager.cc:1905] T e0420148047a43bebfbf096da8573675 P d622c3c03bcf4214904aa23c1ebd7ff9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:41:24.236107 13691 ts_tablet_manager.cc:1918] T e0420148047a43bebfbf096da8573675 P d622c3c03bcf4214904aa23c1ebd7ff9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:41:24.236227 13691 log.cc:1196] T e0420148047a43bebfbf096da8573675 P d622c3c03bcf4214904aa23c1ebd7ff9: Deleting WAL directory at /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311123478858-392-0/minicluster-data/ts-2/wal/wals/e0420148047a43bebfbf096da8573675
I20260925 04:41:24.237994 13689 ts_tablet_manager.cc:1905] T e0420148047a43bebfbf096da8573675 P 8de887c7a4c1454384a914454b9e6457: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:41:24.243160 13689 ts_tablet_manager.cc:1918] T e0420148047a43bebfbf096da8573675 P 8de887c7a4c1454384a914454b9e6457: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260925 04:41:24.243813 13149 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38785 (error 108)
I20260925 04:41:24.245020 13691 ts_tablet_manager.cc:1939] T e0420148047a43bebfbf096da8573675 P d622c3c03bcf4214904aa23c1ebd7ff9: Deleting consensus metadata
I20260925 04:41:24.246609 13150 catalog_manager.cc:4841] TS d622c3c03bcf4214904aa23c1ebd7ff9 (127.0.98.3:33591): tablet e0420148047a43bebfbf096da8573675 (table test-table [id=9c2fa3e875b9487dbad330267e7eb5f6]) successfully deleted
I20260925 04:41:24.246994 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 13365
I20260925 04:41:24.250442 13660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:24.250050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38785 (local address 127.0.0.1:43900)
0925 04:41:24.250260 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:24.250341 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
W20260925 04:41:24.250662 13149 catalog_manager.cc:4584] TS 76ebc568de434d39b3f727e77ed7faea (127.0.98.1:38785): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e0420148047a43bebfbf096da8573675: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38785: connect: Connection refused (error 111)
I20260925 04:41:24.255647 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 13501
I20260925 04:41:24.256000 13660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:24.255388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34677 (local address 127.0.0.1:51760)
0925 04:41:24.255848 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:24.255909 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:34677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364}
I20260925 04:41:24.263015 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 13132
I20260925 04:41:24.276050 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 04:41:24.621927 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 04:41:24.622057 392 mini_ranger.cc:66] Stopping Ranger...
I20260925 04:41:24.957710 392 mini_ranger.cc:68] Stopped Ranger
2026-09-25 04:41:24.957 UTC [12198] LOG: received smart shutdown request
2026-09-25 04:41:24.961 UTC [12198] LOG: background worker "logical replication launcher" (PID 12207) exited with exit code 1
2026-09-25 04:41:24.961 UTC [12202] LOG: shutting down
2026-09-25 04:41:25.145 UTC [12198] LOG: database system is shut down
2026-09-25T04:41:25Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (50038 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:41:25 dist-test-slave-dc95 krb5kdc[13702](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:41:25 dist-test-slave-dc95 krb5kdc[13702](info): set up 2 sockets
Sep 25 04:41:25 dist-test-slave-dc95 krb5kdc[13702](info): commencing operation
krb5kdc: starting...
W20260925 04:41:26.227409 392 mini_kdc.cc:121] Time spent starting KDC: real 0.968s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/krb5kdc/test-user.keytab.
Sep 25 04:41:26 dist-test-slave-dc95 krb5kdc[13702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:41:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:41:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 04:41:26.351177 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45827
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45827,127.0.98.61:37711,127.0.98.60:42049
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:41:26.464637 13715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:41:26.464946 13715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:41:26.465001 13715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:41:26.469256 13715 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:41:26.469338 13715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:41:26.469367 13715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:41:26.469388 13715 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:41:26.469409 13715 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:41:26.475019 13715 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45827,127.0.98.61:37711,127.0.98.60:42049
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:41:26.476282 13715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:41:26.477316 13715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:41:26.485944 13721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:41:26.485944 13720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:41:26.486178 13715 server_base.cc:1034] running on GCE node
W20260925 04:41:26.485949 13723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:41:26.487031 13715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:41:26.488272 13715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:41:26.489470 13715 hybrid_clock.cc:648] HybridClock initialized: now 1790311286489438 us; error 45 us; skew 500 ppm
Sep 25 04:41:26 dist-test-slave-dc95 krb5kdc[13702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311286, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:41:26.492492 13715 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 04:41:26.493413 13715 webserver.cc:460] Webserver started at http://127.0.98.62:40673/ using document root <none> and password file <none>
I20260925 04:41:26.493953 13715 fs_manager.cc:359] Metadata directory not provided
I20260925 04:41:26.494014 13715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:41:26.494191 13715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:41:26.496488 13715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/minicluster-data/master-0/data/instance:
uuid: "229a2e6b7f16481abe3f9bc64f6623ad"
format_stamp: "Formatted at 2026-09-25 04:41:26 on dist-test-slave-dc95"
server_key: "277e704c4b02c0406b0729f6e5b8842b"
server_key_iv: "f721926d39766fce62b5ca3c9775c5b6"
server_key_version: "encryptionkey@0"
I20260925 04:41:26.497069 13715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/minicluster-data/master-0/wal/instance:
uuid: "229a2e6b7f16481abe3f9bc64f6623ad"
format_stamp: "Formatted at 2026-09-25 04:41:26 on dist-test-slave-dc95"
server_key: "277e704c4b02c0406b0729f6e5b8842b"
server_key_iv: "f721926d39766fce62b5ca3c9775c5b6"
server_key_version: "encryptionkey@0"
I20260925 04:41:26.501057 13715 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:41:26.503723 13730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:41:26.505059 13715 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:41:26.505224 13715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/minicluster-data/master-0/wal
uuid: "229a2e6b7f16481abe3f9bc64f6623ad"
format_stamp: "Formatted at 2026-09-25 04:41:26 on dist-test-slave-dc95"
server_key: "277e704c4b02c0406b0729f6e5b8842b"
server_key_iv: "f721926d39766fce62b5ca3c9775c5b6"
server_key_version: "encryptionkey@0"
I20260925 04:41:26.505343 13715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:41:26.550431 13715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:41:26.553735 13715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:41:26.553933 13715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:41:26.561273 13715 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45827
I20260925 04:41:26.561280 13782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45827 every 8 connection(s)
I20260925 04:41:26.562310 13715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
I20260925 04:41:26.565218 13783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:41:26.568477 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 13715
I20260925 04:41:26.568598 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/minicluster-data/master-0/wal/instance
I20260925 04:41:26.568854 392 external_mini_cluster.cc:1428] Setting key 0d545a666128ea6a412d03dccf92ae01
I20260925 04:41:26.569183 13783 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45827 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 25 04:41:26 dist-test-slave-dc95 krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311286, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:41:26.585708 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:26.571596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45827 (local address 127.0.0.1:51334)
0925 04:41:26.572133 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:26.572184 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 04:41:26.573038 (+ 854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:41:26.573813 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:41:26.573817 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:41:26.574342 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:41:26.574716 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:26.574738 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:26.575969 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:26.575973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:26.577149 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:26.577161 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:26.577856 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:26.577859 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:26.577979 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:26.578609 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:41:26.578626 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:41:26.580671 (+ 2045us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:41:26.582890 (+ 2219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:41:26.582897 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:41:26.582907 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:41:26.583132 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:41:26.583456 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:41:26.583458 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:41:26.583459 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:41:26.583558 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:41:26.584297 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:41:26.584303 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:41:26.584437 (+ 134us) client_negotiation.cc:769] Sending connection context
0925 04:41:26.584584 (+ 147us) client_negotiation.cc:240] Negotiation successful
0925 04:41:26.584881 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":1,"spinlock_wait_cycles":41216,"thread_start_us":145,"threads_started":1}
I20260925 04:41:26.585714 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:26.571596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51334 (local address 127.0.98.62:45827)
0925 04:41:26.572055 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0925 04:41:26.572074 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 04:41:26.572295 (+ 221us) server_negotiation.cc:407] Connection header received
0925 04:41:26.573362 (+ 1067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:41:26.573368 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:41:26.573621 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:41:26.573829 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:41:26.574892 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:26.575836 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:26.577281 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:26.577695 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:26.577815 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:26.580832 (+ 3017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:41:26.580837 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:41:26.580848 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:41:26.580869 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:41:26.582766 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:26.583234 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:26.583240 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:26.583242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:26.583371 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:26.583644 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:26.583648 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:26.583650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:26.584177 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:41:26.584312 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 04:41:26.584698 (+ 386us) server_negotiation.cc:299] Negotiation successful
0925 04:41:26.584974 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":86,"server-negotiator.queue_time_us":288,"thread_start_us":124,"threads_started":1}
I20260925 04:41:26.588276 13783 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37711 } has no permanent_uuid. Determining permanent_uuid...
I20260925 04:41:26.589025 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:26.588743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37711 (local address 127.0.0.1:53496)
0925 04:41:26.588856 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:26.588960 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 04:41:26.589260 13731 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37711: connect: Connection refused (error 111)
I20260925 04:41:26.590171 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:26.589898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37711 (local address 127.0.0.1:53504)
0925 04:41:26.590041 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:26.590098 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260925 04:41:26.590319 13783 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37711: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37711: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260925 04:41:26.596832 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:37711
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45827,127.0.98.61:37711,127.0.98.60:42049
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:41:26.639570 13783 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37711 } attempt: 1
I20260925 04:41:26.640501 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:26.640216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37711 (local address 127.0.0.1:53514)
0925 04:41:26.640360 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:26.640422 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 04:41:26.641193 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:26.640980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37711 (local address 127.0.0.1:53528)
0925 04:41:26.641089 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:26.641132 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 04:41:26.641309 13783 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37711: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37711: connect: Connection refused (error 111)
W20260925 04:41:26.708813 13790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:41:26.709110 13790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:41:26.709172 13790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 04:41:26.709476 13783 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37711 } attempt: 2
I20260925 04:41:26.710330 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:26.710034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37711 (local address 127.0.0.1:53538)
0925 04:41:26.710190 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:26.710254 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260925 04:41:26.710960 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:26.710764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37711 (local address 127.0.0.1:53550)
0925 04:41:26.710860 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:26.710904 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 04:41:26.711081 13783 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37711: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37711: connect: Connection refused (error 111)
W20260925 04:41:26.713941 13790 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:41:26.714010 13790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:41:26.714030 13790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:41:26.714046 13790 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:41:26.714061 13790 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:41:26.719702 13790 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45827,127.0.98.61:37711,127.0.98.60:42049
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:37711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:41:26.720842 13790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:41:26.721887 13790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:41:26.730710 13798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:41:26.730661 13795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:41:26.730680 13796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:41:26.730988 13790 server_base.cc:1034] running on GCE node
I20260925 04:41:26.731702 13790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:41:26.732949 13790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:41:26.734238 13790 hybrid_clock.cc:648] HybridClock initialized: now 1790311286734201 us; error 51 us; skew 500 ppm
Sep 25 04:41:26 dist-test-slave-dc95 krb5kdc[13702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311286, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:41:26.737211 13790 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260925 04:41:26.738159 13790 webserver.cc:460] Webserver started at http://127.0.98.61:33999/ using document root <none> and password file <none>
I20260925 04:41:26.738758 13790 fs_manager.cc:359] Metadata directory not provided
I20260925 04:41:26.738829 13790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:41:26.739015 13790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:41:26.741145 13790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/minicluster-data/master-1/data/instance:
uuid: "1efed41a4dce415180a0a488b19936bf"
format_stamp: "Formatted at 2026-09-25 04:41:26 on dist-test-slave-dc95"
server_key: "675662d311d8bb0dab7b62a18d493e04"
server_key_iv: "8924daa57bf6710e196876e27b86c2c8"
server_key_version: "encryptionkey@0"
I20260925 04:41:26.741706 13790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/minicluster-data/master-1/wal/instance:
uuid: "1efed41a4dce415180a0a488b19936bf"
format_stamp: "Formatted at 2026-09-25 04:41:26 on dist-test-slave-dc95"
server_key: "675662d311d8bb0dab7b62a18d493e04"
server_key_iv: "8924daa57bf6710e196876e27b86c2c8"
server_key_version: "encryptionkey@0"
I20260925 04:41:26.745548 13790 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 04:41:26.748227 13805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:41:26.749718 13790 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:41:26.749866 13790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/minicluster-data/master-1/wal
uuid: "1efed41a4dce415180a0a488b19936bf"
format_stamp: "Formatted at 2026-09-25 04:41:26 on dist-test-slave-dc95"
server_key: "675662d311d8bb0dab7b62a18d493e04"
server_key_iv: "8924daa57bf6710e196876e27b86c2c8"
server_key_version: "encryptionkey@0"
I20260925 04:41:26.749982 13790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:41:26.769359 13790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:41:26.773092 13790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:41:26.773299 13790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:41:26.780334 13790 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:37711
I20260925 04:41:26.780437 13857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:37711 every 8 connection(s)
I20260925 04:41:26.781383 13790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/minicluster-data/master-1/data/info.pb
I20260925 04:41:26.783288 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 13790
I20260925 04:41:26.783392 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/minicluster-data/master-1/wal/instance
I20260925 04:41:26.783689 392 external_mini_cluster.cc:1428] Setting key 4d7c48f93bf291278151488ba763142e
I20260925 04:41:26.784403 13858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:41:26.788625 13858 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45827 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 25 04:41:26 dist-test-slave-dc95 krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311286, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:41:26.802250 13783 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37711 } attempt: 3
I20260925 04:41:26.803886 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:26.790697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51340 (local address 127.0.98.62:45827)
0925 04:41:26.790827 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 04:41:26.790830 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:41:26.792041 (+ 1211us) server_negotiation.cc:407] Connection header received
0925 04:41:26.793047 (+ 1006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:41:26.793051 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:41:26.793119 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:41:26.793198 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:41:26.794456 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:26.795045 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:26.796313 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:26.796553 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:26.796644 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:26.799413 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:41:26.799416 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:41:26.799418 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:41:26.799448 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:41:26.801272 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:26.801975 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:26.801978 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:26.801980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:26.802068 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:26.802512 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:26.802515 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:26.802517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:26.802758 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:41:26.802856 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 04:41:26.803550 (+ 694us) server_negotiation.cc:299] Negotiation successful
0925 04:41:26.803722 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 04:41:26.804793 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:26.791260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45827 (local address 127.0.0.1:51340)
0925 04:41:26.791877 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:26.791923 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 04:41:26.792814 (+ 891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:41:26.793381 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:41:26.793388 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:41:26.793747 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:41:26.794299 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:26.794313 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:26.795171 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:26.795174 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:26.796184 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:26.796195 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:26.796719 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:26.796723 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:26.796918 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:26.797590 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:41:26.797611 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:41:26.799208 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:41:26.801431 (+ 2223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:41:26.801439 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:41:26.801458 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:41:26.801846 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:41:26.802213 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:41:26.802215 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:41:26.802217 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:41:26.802385 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:41:26.802910 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:41:26.802917 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:41:26.803282 (+ 365us) client_negotiation.cc:769] Sending connection context
0925 04:41:26.803470 (+ 188us) client_negotiation.cc:240] Negotiation successful
0925 04:41:26.803894 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":151,"threads_started":1}
I20260925 04:41:26.806457 13858 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37711 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 04:41:26 dist-test-slave-dc95 krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311286, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 25 04:41:26 dist-test-slave-dc95 krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311286, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260925 04:41:26.814481 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:26.802762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37711 (local address 127.0.0.1:53566)
0925 04:41:26.802964 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:26.802979 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:41:26.803061 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:41:26.803766 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:41:26.803769 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:41:26.803935 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:41:26.804468 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:26.804477 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:26.806018 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:26.806022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:26.806828 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:26.806838 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:26.807578 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:26.807581 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:26.807791 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:26.808323 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:41:26.808339 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:41:26.809784 (+ 1445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:41:26.812047 (+ 2263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:41:26.812051 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:41:26.812073 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:41:26.812315 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:41:26.813213 (+ 898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:41:26.813217 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:41:26.813219 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:41:26.813339 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:41:26.813933 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:41:26.813936 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:41:26.814056 (+ 120us) client_negotiation.cc:769] Sending connection context
0925 04:41:26.814178 (+ 122us) client_negotiation.cc:240] Negotiation successful
0925 04:41:26.814340 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125}
I20260925 04:41:26.814731 13863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:26.802904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53566 (local address 127.0.98.61:37711)
0925 04:41:26.803330 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0925 04:41:26.803344 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 04:41:26.803368 (+ 24us) server_negotiation.cc:407] Connection header received
0925 04:41:26.803414 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:41:26.803420 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:41:26.803597 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:41:26.803776 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:41:26.804590 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:26.805883 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:26.806959 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:26.807471 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:26.807577 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:26.809956 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:41:26.809966 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:41:26.809981 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:41:26.810005 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:41:26.811936 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:26.812414 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:26.812421 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:26.812423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:26.813036 (+ 613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:26.813438 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:26.813442 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:26.813444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:26.813806 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:41:26.813950 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 04:41:26.814320 (+ 370us) server_negotiation.cc:299] Negotiation successful
0925 04:41:26.814512 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":164,"threads_started":1}
I20260925 04:41:26.814953 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:42049
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45827,127.0.98.61:37711,127.0.98.60:42049
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:41:26.816221 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:26.807043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37711 (local address 127.0.0.1:53570)
0925 04:41:26.807296 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:26.807311 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:41:26.807410 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:41:26.807866 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:41:26.807869 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:41:26.808057 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:41:26.808157 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:26.808164 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:26.809089 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:26.809092 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:26.809678 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:26.809687 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:26.810212 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:26.810216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:26.810310 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:26.810652 (+ 342us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:41:26.810666 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:41:26.812086 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:41:26.814659 (+ 2573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:41:26.814663 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:41:26.814665 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:41:26.814983 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:41:26.815261 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:41:26.815264 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:41:26.815266 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:41:26.815336 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:41:26.815760 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:41:26.815763 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:41:26.815837 (+ 74us) client_negotiation.cc:769] Sending connection context
0925 04:41:26.815920 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 04:41:26.816066 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104}
I20260925 04:41:26.816355 13864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:26.807317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53570 (local address 127.0.98.61:37711)
0925 04:41:26.807660 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0925 04:41:26.807670 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 04:41:26.807683 (+ 13us) server_negotiation.cc:407] Connection header received
0925 04:41:26.807715 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:41:26.807717 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:41:26.807756 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:41:26.807822 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:41:26.808252 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:26.808994 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:26.809776 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:26.810108 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:26.810195 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:26.812216 (+ 2021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:41:26.812219 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:41:26.812222 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:41:26.812246 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:41:26.813812 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:26.815084 (+ 1272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:26.815087 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:26.815089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:26.815179 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:26.815414 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:26.815417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:26.815418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:26.815666 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:41:26.815754 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 04:41:26.815971 (+ 217us) server_negotiation.cc:299] Negotiation successful
0925 04:41:26.816190 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":242,"thread_start_us":128,"threads_started":1}
I20260925 04:41:26.816960 13858 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42049 } has no permanent_uuid. Determining permanent_uuid...
I20260925 04:41:26.817122 13783 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42049 } has no permanent_uuid. Determining permanent_uuid...
I20260925 04:41:26.817696 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:26.817417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42049 (local address 127.0.0.1:54558)
0925 04:41:26.817563 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:26.817628 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260925 04:41:26.817866 13807 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42049: connect: Connection refused (error 111)
I20260925 04:41:26.817920 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:26.817663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42049 (local address 127.0.0.1:54564)
0925 04:41:26.817793 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:26.817848 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 04:41:26.818895 13867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:26.818368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42049 (local address 127.0.0.1:54572)
0925 04:41:26.818745 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:26.818798 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":90,"threads_started":1}
I20260925 04:41:26.818900 13868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:26.818528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42049 (local address 127.0.0.1:54580)
0925 04:41:26.818766 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:26.818815 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":84,"threads_started":1}
W20260925 04:41:26.819092 13783 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42049: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42049: connect: Connection refused (error 111)
W20260925 04:41:26.819098 13858 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42049: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42049: connect: Connection refused (error 111)
I20260925 04:41:26.850277 13783 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42049 } attempt: 1
I20260925 04:41:26.851166 13867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:26.850865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42049 (local address 127.0.0.1:54596)
0925 04:41:26.851013 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:26.851076 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 04:41:26.851845 13867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:26.851654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42049 (local address 127.0.0.1:54612)
0925 04:41:26.851743 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:26.851788 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260925 04:41:26.851958 13783 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42049: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42049: connect: Connection refused (error 111)
I20260925 04:41:26.868381 13858 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42049 } attempt: 1
I20260925 04:41:26.869310 13868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:26.868985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42049 (local address 127.0.0.1:54624)
0925 04:41:26.869138 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:26.869220 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260925 04:41:26.870250 13868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:26.869937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42049 (local address 127.0.0.1:54640)
0925 04:41:26.870110 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:26.870169 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260925 04:41:26.870435 13858 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42049: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42049: connect: Connection refused (error 111)
I20260925 04:41:26.927183 13783 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42049 } attempt: 2
I20260925 04:41:26.928076 13867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:26.927780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42049 (local address 127.0.0.1:54642)
0925 04:41:26.927933 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:26.927994 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260925 04:41:26.928901 13867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:26.928625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42049 (local address 127.0.0.1:54656)
0925 04:41:26.928781 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:26.928831 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
W20260925 04:41:26.929062 13783 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42049: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42049: connect: Connection refused (error 111)
W20260925 04:41:26.933633 13865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:41:26.933926 13865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:41:26.933983 13865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:41:26.938267 13865 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:41:26.938356 13865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:41:26.938385 13865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:41:26.938409 13865 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:41:26.938437 13865 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:41:26.938607 13858 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42049 } attempt: 2
I20260925 04:41:26.939481 13868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:26.939162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42049 (local address 127.0.0.1:54668)
0925 04:41:26.939336 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:26.939401 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260925 04:41:26.940248 13868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:26.940040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42049 (local address 127.0.0.1:54680)
0925 04:41:26.940142 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:26.940188 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 04:41:26.940377 13858 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42049: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42049: connect: Connection refused (error 111)
I20260925 04:41:26.944511 13865 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45827,127.0.98.61:37711,127.0.98.60:42049
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:42049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:41:26.945639 13865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:41:26.946693 13865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 04:41:26.955685 13865 server_base.cc:1034] running on GCE node
W20260925 04:41:26.955593 13874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:41:26.955593 13876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:41:26.955593 13873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:41:26.956496 13865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:41:26.957708 13865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:41:26.958902 13865 hybrid_clock.cc:648] HybridClock initialized: now 1790311286958882 us; error 35 us; skew 500 ppm
Sep 25 04:41:26 dist-test-slave-dc95 krb5kdc[13702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311286, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:41:26.961851 13865 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260925 04:41:26.962843 13865 webserver.cc:460] Webserver started at http://127.0.98.60:35511/ using document root <none> and password file <none>
I20260925 04:41:26.963454 13865 fs_manager.cc:359] Metadata directory not provided
I20260925 04:41:26.963524 13865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:41:26.963776 13865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:41:26.965942 13865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/minicluster-data/master-2/data/instance:
uuid: "2bf8f8f99fd5489798840d6d1b7f4e0f"
format_stamp: "Formatted at 2026-09-25 04:41:26 on dist-test-slave-dc95"
server_key: "41c7150a8f7611af040d933c178e83cf"
server_key_iv: "bda88cf0b29bc72af7b7895aaf190e02"
server_key_version: "encryptionkey@0"
I20260925 04:41:26.966562 13865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/minicluster-data/master-2/wal/instance:
uuid: "2bf8f8f99fd5489798840d6d1b7f4e0f"
format_stamp: "Formatted at 2026-09-25 04:41:26 on dist-test-slave-dc95"
server_key: "41c7150a8f7611af040d933c178e83cf"
server_key_iv: "bda88cf0b29bc72af7b7895aaf190e02"
server_key_version: "encryptionkey@0"
I20260925 04:41:26.970744 13865 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 04:41:26.973690 13883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:41:26.975283 13865 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:41:26.975433 13865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/minicluster-data/master-2/wal
uuid: "2bf8f8f99fd5489798840d6d1b7f4e0f"
format_stamp: "Formatted at 2026-09-25 04:41:26 on dist-test-slave-dc95"
server_key: "41c7150a8f7611af040d933c178e83cf"
server_key_iv: "bda88cf0b29bc72af7b7895aaf190e02"
server_key_version: "encryptionkey@0"
I20260925 04:41:26.975556 13865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:41:27.013947 13865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:41:27.017297 13865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:41:27.017488 13865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:41:27.024427 13865 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:42049
I20260925 04:41:27.024447 13935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:42049 every 8 connection(s)
I20260925 04:41:27.025454 13865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/minicluster-data/master-2/data/info.pb
I20260925 04:41:27.028312 13936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:41:27.028299 13783 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42049 } attempt: 3
I20260925 04:41:27.031560 13858 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42049 } attempt: 3
I20260925 04:41:27.034021 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 13865
I20260925 04:41:27.034133 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311123478858-392-0/minicluster-data/master-2/wal/instance
I20260925 04:41:27.033744 13936 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45827 } has no permanent_uuid. Determining permanent_uuid...
I20260925 04:41:27.034423 392 external_mini_cluster.cc:1428] Setting key 6bed3f20a55c3b852e27b9163da4a9e5
Sep 25 04:41:27 dist-test-slave-dc95 krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311286, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 25 04:41:27 dist-test-slave-dc95 krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311286, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 25 04:41:27 dist-test-slave-dc95 krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 04:41:27.043004 13867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:27.028797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42049 (local address 127.0.0.1:54696)
0925 04:41:27.028932 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:27.028947 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:41:27.029045 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:41:27.031300 (+ 2255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:41:27.031304 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:41:27.031562 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:41:27.031738 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:27.031748 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:27.033615 (+ 1867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:27.033619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:27.034486 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:27.034500 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:27.035538 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:27.035543 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:27.035674 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:27.036403 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:41:27.036418 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:41:27.038429 (+ 2011us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:41:27.040884 (+ 2455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:41:27.040888 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:41:27.040891 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:41:27.041165 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:41:27.041569 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:41:27.041573 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:41:27.041575 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:41:27.041670 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:41:27.042463 (+ 793us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:41:27.042466 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:41:27.042576 (+ 110us) client_negotiation.cc:769] Sending connection context
0925 04:41:27.042660 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 04:41:27.042809 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
Sep 25 04:41:27 dist-test-slave-dc95 krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311286, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:41:27.043372 13868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:27.032048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42049 (local address 127.0.0.1:54708)
0925 04:41:27.032244 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:27.032260 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:41:27.032359 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:41:27.032723 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:41:27.032727 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:41:27.032942 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:41:27.033070 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:27.033076 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:27.033950 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:27.033954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:27.034746 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:27.034757 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:27.035577 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:27.035581 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:27.035700 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:27.036276 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:41:27.036292 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:41:27.038617 (+ 2325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:41:27.041716 (+ 3099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:41:27.041719 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:41:27.041722 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:41:27.041984 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:41:27.042298 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:41:27.042301 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:41:27.042303 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:41:27.042392 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:41:27.042823 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:41:27.042826 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:41:27.042929 (+ 103us) client_negotiation.cc:769] Sending connection context
0925 04:41:27.043007 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 04:41:27.043198 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122}
I20260925 04:41:27.045003 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:27.029479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54696 (local address 127.0.98.60:42049)
0925 04:41:27.029874 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0925 04:41:27.029889 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 04:41:27.029925 (+ 36us) server_negotiation.cc:407] Connection header received
0925 04:41:27.030630 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:41:27.030647 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:41:27.030994 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:41:27.031322 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:41:27.031943 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:27.033450 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:27.034637 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:27.035368 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:27.036914 (+ 1546us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:27.038598 (+ 1684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:41:27.038605 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:41:27.038617 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:41:27.038646 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:41:27.040761 (+ 2115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:27.041286 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:27.041293 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:27.041299 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:27.041445 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:27.041781 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:27.041784 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:27.041785 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:27.042338 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:41:27.043372 (+ 1034us) server_negotiation.cc:1035] Waiting for connection context
0925 04:41:27.043540 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 04:41:27.043913 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":260,"spinlock_wait_cycles":4992,"thread_start_us":119,"threads_started":1}
I20260925 04:41:27.045001 13939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:27.032125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54708 (local address 127.0.98.60:42049)
0925 04:41:27.032508 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0925 04:41:27.032515 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:41:27.032525 (+ 10us) server_negotiation.cc:407] Connection header received
0925 04:41:27.032552 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:41:27.032554 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:41:27.032586 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:41:27.032666 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:41:27.033191 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:27.033815 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:27.035000 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:27.035461 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:27.035981 (+ 520us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:27.038762 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:41:27.038778 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:41:27.038780 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:41:27.038804 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:41:27.041590 (+ 2786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:27.042105 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:27.042109 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:27.042110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:27.042181 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:27.042515 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:27.042518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:27.042520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:27.042712 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:41:27.043644 (+ 932us) server_negotiation.cc:1035] Waiting for connection context
0925 04:41:27.043771 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 04:41:27.043996 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":211,"server-negotiator.queue_time_us":324,"thread_start_us":168,"threads_started":1}
I20260925 04:41:27.047935 13941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:27.035407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54718 (local address 127.0.98.60:42049)
0925 04:41:27.035759 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0925 04:41:27.035768 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:41:27.035780 (+ 12us) server_negotiation.cc:407] Connection header received
0925 04:41:27.035810 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:41:27.035813 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:41:27.035853 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:41:27.036788 (+ 935us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:41:27.037453 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:27.038289 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:27.039456 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:27.039800 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:27.039929 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:27.043371 (+ 3442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:41:27.043374 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:41:27.043377 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:41:27.043401 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:41:27.045737 (+ 2336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:27.046136 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:27.046139 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:27.046141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:27.046217 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:27.046461 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:27.046464 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:27.046466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:27.046638 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:41:27.046797 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 04:41:27.047655 (+ 858us) server_negotiation.cc:299] Negotiation successful
0925 04:41:27.047769 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":96,"threads_started":1}
I20260925 04:41:27.049255 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:27.036845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51342 (local address 127.0.98.62:45827)
0925 04:41:27.036998 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 04:41:27.037002 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:41:27.037313 (+ 311us) server_negotiation.cc:407] Connection header received
0925 04:41:27.037435 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:41:27.037437 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:41:27.037477 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:41:27.037567 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:41:27.038324 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:27.039409 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:27.040444 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:27.040772 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:27.040868 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:27.044430 (+ 3562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:41:27.044434 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:41:27.044436 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:41:27.044462 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:41:27.047234 (+ 2772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:27.047790 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:27.047793 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:27.047795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:27.047878 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:27.048363 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:27.048367 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:27.048368 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:27.048570 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:41:27.048654 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 04:41:27.048998 (+ 344us) server_negotiation.cc:299] Negotiation successful
0925 04:41:27.049105 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 04:41:27.049268 13942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:27.036019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45827 (local address 127.0.0.1:51342)
0925 04:41:27.037153 (+ 1134us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:27.037208 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 04:41:27.037350 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:41:27.037729 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:41:27.037733 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:41:27.038057 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:41:27.038213 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:27.038220 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:27.039536 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:27.039539 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:27.040324 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:27.040333 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:27.041303 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:27.041307 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:27.041426 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:27.041963 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:41:27.041977 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:41:27.044267 (+ 2290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:41:27.047342 (+ 3075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:41:27.047348 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:41:27.047354 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:41:27.047679 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:41:27.048174 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:41:27.048176 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:41:27.048177 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:41:27.048252 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:41:27.048683 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:41:27.048690 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:41:27.048797 (+ 107us) client_negotiation.cc:769] Sending connection context
0925 04:41:27.048909 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 04:41:27.049092 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":970,"spinlock_wait_cycles":4352,"thread_start_us":89,"threads_started":1}
I20260925 04:41:27.049772 13858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1efed41a4dce415180a0a488b19936bf: Bootstrap starting.
I20260925 04:41:27.049772 13783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 229a2e6b7f16481abe3f9bc64f6623ad: Bootstrap starting.
I20260925 04:41:27.052639 13858 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1efed41a4dce415180a0a488b19936bf: Neither blocks nor log segments found. Creating new log.
I20260925 04:41:27.052922 13936 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37711 } has no permanent_uuid. Determining permanent_uuid...
I20260925 04:41:27.053552 13858 log.cc:824] T 00000000000000000000000000000000 P 1efed41a4dce415180a0a488b19936bf: Log is configured to *not* fsync() on all Append() calls
I20260925 04:41:27.053647 13783 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 229a2e6b7f16481abe3f9bc64f6623ad: Neither blocks nor log segments found. Creating new log.
I20260925 04:41:27.054684 13783 log.cc:824] T 00000000000000000000000000000000 P 229a2e6b7f16481abe3f9bc64f6623ad: Log is configured to *not* fsync() on all Append() calls
I20260925 04:41:27.056133 13858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1efed41a4dce415180a0a488b19936bf: No bootstrap required, opened a new log
I20260925 04:41:27.056700 13783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 229a2e6b7f16481abe3f9bc64f6623ad: No bootstrap required, opened a new log
Sep 25 04:41:27 dist-test-slave-dc95 krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 25 04:41:27 dist-test-slave-dc95 krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311286, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 04:41:27.060918 13783 raft_consensus.cc:348] T 00000000000000000000000000000000 P 229a2e6b7f16481abe3f9bc64f6623ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "229a2e6b7f16481abe3f9bc64f6623ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45827 } } peers { permanent_uuid: "1efed41a4dce415180a0a488b19936bf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37711 } } peers { permanent_uuid: "2bf8f8f99fd5489798840d6d1b7f4e0f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42049 } }
I20260925 04:41:27.061172 13783 raft_consensus.cc:374] T 00000000000000000000000000000000 P 229a2e6b7f16481abe3f9bc64f6623ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:41:27.061228 13783 raft_consensus.cc:729] T 00000000000000000000000000000000 P 229a2e6b7f16481abe3f9bc64f6623ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 229a2e6b7f16481abe3f9bc64f6623ad, State: Initialized, Role: FOLLOWER
I20260925 04:41:27.061893 13783 consensus_queue.cc:260] T 00000000000000000000000000000000 P 229a2e6b7f16481abe3f9bc64f6623ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "229a2e6b7f16481abe3f9bc64f6623ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45827 } } peers { permanent_uuid: "1efed41a4dce415180a0a488b19936bf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37711 } } peers { permanent_uuid: "2bf8f8f99fd5489798840d6d1b7f4e0f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42049 } }
I20260925 04:41:27.062256 13858 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1efed41a4dce415180a0a488b19936bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "229a2e6b7f16481abe3f9bc64f6623ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45827 } } peers { permanent_uuid: "1efed41a4dce415180a0a488b19936bf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37711 } } peers { permanent_uuid: "2bf8f8f99fd5489798840d6d1b7f4e0f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42049 } }
I20260925 04:41:27.062484 13858 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1efed41a4dce415180a0a488b19936bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:41:27.062543 13858 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1efed41a4dce415180a0a488b19936bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1efed41a4dce415180a0a488b19936bf, State: Initialized, Role: FOLLOWER
I20260925 04:41:27.063064 13858 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1efed41a4dce415180a0a488b19936bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "229a2e6b7f16481abe3f9bc64f6623ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45827 } } peers { permanent_uuid: "1efed41a4dce415180a0a488b19936bf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37711 } } peers { permanent_uuid: "2bf8f8f99fd5489798840d6d1b7f4e0f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42049 } }
I20260925 04:41:27.063424 13783 sys_catalog.cc:564] T 00000000000000000000000000000000 P 229a2e6b7f16481abe3f9bc64f6623ad [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:41:27.064082 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:27.052959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51352 (local address 127.0.98.62:45827)
0925 04:41:27.053172 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 04:41:27.053176 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:41:27.053189 (+ 13us) server_negotiation.cc:407] Connection header received
0925 04:41:27.053226 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:41:27.053229 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:41:27.053273 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:41:27.053336 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:41:27.054317 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:27.055026 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:27.056318 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:27.056629 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:27.056723 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:27.060243 (+ 3520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:41:27.060247 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:41:27.060251 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:41:27.060278 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:41:27.061781 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:27.062282 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:27.062286 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:27.062288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:27.062374 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:27.062658 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:27.062661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:27.062663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:27.062884 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:41:27.063057 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 04:41:27.063802 (+ 745us) server_negotiation.cc:299] Negotiation successful
0925 04:41:27.063929 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139}
I20260925 04:41:27.064409 13863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:27.053631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53572 (local address 127.0.98.61:37711)
0925 04:41:27.053758 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 04:41:27.053762 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:41:27.054391 (+ 629us) server_negotiation.cc:407] Connection header received
0925 04:41:27.054587 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:41:27.054591 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:41:27.054649 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:41:27.054700 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:41:27.055408 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:27.056143 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:27.057192 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:27.057524 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:27.057596 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:27.060233 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:41:27.060237 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:41:27.060240 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:41:27.060272 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:41:27.061954 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:27.062911 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:27.062914 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:27.062916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:27.063000 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:27.063381 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:27.063384 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:27.063386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:27.063687 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:41:27.063758 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 04:41:27.064150 (+ 392us) server_negotiation.cc:299] Negotiation successful
0925 04:41:27.064273 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 04:41:27.064494 13858 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1efed41a4dce415180a0a488b19936bf [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:41:27.065945 13945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 229a2e6b7f16481abe3f9bc64f6623ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "229a2e6b7f16481abe3f9bc64f6623ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45827 } } peers { permanent_uuid: "1efed41a4dce415180a0a488b19936bf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37711 } } peers { permanent_uuid: "2bf8f8f99fd5489798840d6d1b7f4e0f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42049 } } }
I20260925 04:41:27.066059 13945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 229a2e6b7f16481abe3f9bc64f6623ad [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:41:27.066272 13946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1efed41a4dce415180a0a488b19936bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "229a2e6b7f16481abe3f9bc64f6623ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45827 } } peers { permanent_uuid: "1efed41a4dce415180a0a488b19936bf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37711 } } peers { permanent_uuid: "2bf8f8f99fd5489798840d6d1b7f4e0f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42049 } } }
I20260925 04:41:27.066404 13946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1efed41a4dce415180a0a488b19936bf [sys.catalog]: This master's current role is: FOLLOWER
W20260925 04:41:27.069684 13959 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 229a2e6b7f16481abe3f9bc64f6623ad: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 04:41:27.069684 13968 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1efed41a4dce415180a0a488b19936bf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 04:41:27.069777 13968 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260925 04:41:27.069777 13959 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 25 04:41:27 dist-test-slave-dc95 krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 04:41:27.073704 13942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:27.053367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37711 (local address 127.0.0.1:53572)
0925 04:41:27.054316 (+ 949us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:27.054330 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:41:27.054459 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:41:27.054835 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:41:27.054838 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:41:27.055020 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:41:27.055139 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:27.055146 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:27.056280 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:27.056283 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:27.057052 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:27.057063 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:27.057764 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:27.057768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:27.057869 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:27.058289 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:41:27.058302 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:41:27.059992 (+ 1690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:41:27.062409 (+ 2417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:41:27.062413 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:41:27.062415 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:41:27.062800 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:41:27.063106 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:41:27.063109 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:41:27.063111 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:41:27.063199 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:41:27.063866 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:41:27.063869 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:41:27.063974 (+ 105us) client_negotiation.cc:769] Sending connection context
0925 04:41:27.071314 (+ 7340us) client_negotiation.cc:240] Negotiation successful
0925 04:41:27.073511 (+ 2197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":875}
I20260925 04:41:27.074397 13936 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42049 } has no permanent_uuid. Determining permanent_uuid...
I20260925 04:41:27.072398 13715 master_runner.cc:186] Error getting master registration for 127.0.98.60:42049: OK, instance_id { permanent_uuid: "2bf8f8f99fd5489798840d6d1b7f4e0f" instance_seqno: 1790311287022526 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 04:41:27.077791 13715 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 04:41:27.078253 13946 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1efed41a4dce415180a0a488b19936bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:41:27.078423 13946 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1efed41a4dce415180a0a488b19936bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "229a2e6b7f16481abe3f9bc64f6623ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45827 } } peers { permanent_uuid: "1efed41a4dce415180a0a488b19936bf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37711 } } peers { permanent_uuid: "2bf8f8f99fd5489798840d6d1b7f4e0f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42049 } }
I20260925 04:41:27.078974 13863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:27.067427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53582 (local address 127.0.98.61:37711)
0925 04:41:27.067644 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 04:41:27.067648 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:41:27.067662 (+ 14us) server_negotiation.cc:407] Connection header received
0925 04:41:27.067711 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:41:27.067713 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:41:27.067755 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:41:27.067829 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:41:27.068698 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:27.069415 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:27.070501 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:27.070788 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:27.070874 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:27.073537 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:41:27.073541 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:41:27.073544 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:41:27.073572 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:41:27.076003 (+ 2431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:27.076589 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:27.076592 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:27.076594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:27.076687 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:27.077000 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:27.077003 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:27.077004 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:27.077237 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:41:27.078506 (+ 1269us) server_negotiation.cc:1035] Waiting for connection context
0925 04:41:27.078671 (+ 165us) server_negotiation.cc:299] Negotiation successful
0925 04:41:27.078814 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260925 04:41:27.079520 13946 leader_election.cc:288] T 00000000000000000000000000000000 P 1efed41a4dce415180a0a488b19936bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 229a2e6b7f16481abe3f9bc64f6623ad (127.0.98.62:45827), 2bf8f8f99fd5489798840d6d1b7f4e0f (127.0.98.60:42049)
Sep 25 04:41:27 dist-test-slave-dc95 krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311286, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 04:41:27.080519 13757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1efed41a4dce415180a0a488b19936bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "229a2e6b7f16481abe3f9bc64f6623ad" is_pre_election: true
I20260925 04:41:27.080904 13757 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 229a2e6b7f16481abe3f9bc64f6623ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1efed41a4dce415180a0a488b19936bf in term 0.
I20260925 04:41:27.081851 13808 leader_election.cc:302] T 00000000000000000000000000000000 P 1efed41a4dce415180a0a488b19936bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1efed41a4dce415180a0a488b19936bf, 229a2e6b7f16481abe3f9bc64f6623ad; no voters:
I20260925 04:41:27.082237 13946 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1efed41a4dce415180a0a488b19936bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:41:27.082325 13946 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1efed41a4dce415180a0a488b19936bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:41:27.082367 13946 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1efed41a4dce415180a0a488b19936bf [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:41:27.083703 13946 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1efed41a4dce415180a0a488b19936bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "229a2e6b7f16481abe3f9bc64f6623ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45827 } } peers { permanent_uuid: "1efed41a4dce415180a0a488b19936bf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37711 } } peers { permanent_uuid: "2bf8f8f99fd5489798840d6d1b7f4e0f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42049 } }
I20260925 04:41:27.084694 13946 leader_election.cc:288] T 00000000000000000000000000000000 P 1efed41a4dce415180a0a488b19936bf [CANDIDATE]: Term 1 election: Requested vote from peers 229a2e6b7f16481abe3f9bc64f6623ad (127.0.98.62:45827), 2bf8f8f99fd5489798840d6d1b7f4e0f (127.0.98.60:42049)
I20260925 04:41:27.085276 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:41:27.085456 13942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:27.074962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42049 (local address 127.0.0.1:54720)
0925 04:41:27.075101 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:27.075114 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:41:27.075242 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:41:27.075565 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:41:27.075568 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:41:27.075783 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:41:27.075891 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:27.075898 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:27.076845 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:27.076848 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:27.077585 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:27.077596 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:27.078229 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:27.078232 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:27.078304 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:27.078854 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:41:27.078868 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:41:27.080482 (+ 1614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:41:27.083722 (+ 3240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:41:27.083726 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:41:27.083729 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:41:27.083992 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:41:27.084406 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:41:27.084409 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:41:27.084411 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:41:27.084494 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:41:27.084902 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:41:27.084904 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:41:27.085033 (+ 129us) client_negotiation.cc:769] Sending connection context
0925 04:41:27.085124 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 04:41:27.085284 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260925 04:41:27.085901 13941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:27.075195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54720 (local address 127.0.98.60:42049)
0925 04:41:27.075301 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0925 04:41:27.075305 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:41:27.075317 (+ 12us) server_negotiation.cc:407] Connection header received
0925 04:41:27.075346 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:41:27.075348 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:41:27.075387 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:41:27.075452 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:41:27.076012 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:27.076749 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:27.077704 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:27.078028 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:27.078118 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:27.080613 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:41:27.080617 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:41:27.080619 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:41:27.080645 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:41:27.082472 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:27.084212 (+ 1740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:27.084216 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:27.084218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:27.084320 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:27.084580 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:27.084583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:27.084584 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:27.084821 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:41:27.084914 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 04:41:27.085229 (+ 315us) server_negotiation.cc:299] Negotiation successful
0925 04:41:27.085442 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 04:41:27.087924 13757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1efed41a4dce415180a0a488b19936bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "229a2e6b7f16481abe3f9bc64f6623ad"
I20260925 04:41:27.088100 13757 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 229a2e6b7f16481abe3f9bc64f6623ad [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:41:27.089148 13936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bf8f8f99fd5489798840d6d1b7f4e0f: Bootstrap starting.
I20260925 04:41:27.089586 13757 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 229a2e6b7f16481abe3f9bc64f6623ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1efed41a4dce415180a0a488b19936bf in term 1.
I20260925 04:41:27.076591 13790 master_runner.cc:186] Error getting master registration for 127.0.98.60:42049: OK, instance_id { permanent_uuid: "2bf8f8f99fd5489798840d6d1b7f4e0f" instance_seqno: 1790311287022526 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 04:41:27.090197 13790 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 04:41:27.091779 13936 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2bf8f8f99fd5489798840d6d1b7f4e0f: Neither blocks nor log segments found. Creating new log.
I20260925 04:41:27.091926 13808 leader_election.cc:302] T 00000000000000000000000000000000 P 1efed41a4dce415180a0a488b19936bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1efed41a4dce415180a0a488b19936bf, 229a2e6b7f16481abe3f9bc64f6623ad; no voters:
I20260925 04:41:27.092675 13946 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1efed41a4dce415180a0a488b19936bf [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:41:27.092909 13946 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1efed41a4dce415180a0a488b19936bf [term 1 LEADER]: Becoming Leader. State: Replica: 1efed41a4dce415180a0a488b19936bf, State: Running, Role: LEADER
I20260925 04:41:27.093297 13946 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1efed41a4dce415180a0a488b19936bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "229a2e6b7f16481abe3f9bc64f6623ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45827 } } peers { permanent_uuid: "1efed41a4dce415180a0a488b19936bf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37711 } } peers { permanent_uuid: "2bf8f8f99fd5489798840d6d1b7f4e0f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42049 } }
I20260925 04:41:27.095920 13936 log.cc:824] T 00000000000000000000000000000000 P 2bf8f8f99fd5489798840d6d1b7f4e0f: Log is configured to *not* fsync() on all Append() calls
I20260925 04:41:27.097118 13977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1efed41a4dce415180a0a488b19936bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1efed41a4dce415180a0a488b19936bf. Latest consensus state: current_term: 1 leader_uuid: "1efed41a4dce415180a0a488b19936bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "229a2e6b7f16481abe3f9bc64f6623ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45827 } } peers { permanent_uuid: "1efed41a4dce415180a0a488b19936bf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37711 } } peers { permanent_uuid: "2bf8f8f99fd5489798840d6d1b7f4e0f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42049 } } }
I20260925 04:41:27.097301 13977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1efed41a4dce415180a0a488b19936bf [sys.catalog]: This master's current role is: LEADER
I20260925 04:41:27.097735 13980 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:41:27.098767 13980 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:41:27.100687 13863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:27.091778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53598 (local address 127.0.98.61:37711)
0925 04:41:27.092182 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0925 04:41:27.092186 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:41:27.092200 (+ 14us) server_negotiation.cc:407] Connection header received
0925 04:41:27.092236 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:41:27.092239 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:41:27.092286 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:41:27.092362 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:41:27.093418 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:27.094088 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:27.095252 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:27.095544 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:27.095640 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:27.097269 (+ 1629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:41:27.097273 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:41:27.097275 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:41:27.097301 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:41:27.098799 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:27.099341 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:27.099345 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:27.099347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:27.099436 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:27.099757 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:27.099760 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:27.099762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:27.100005 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:41:27.100140 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 04:41:27.100417 (+ 277us) server_negotiation.cc:299] Negotiation successful
0925 04:41:27.100525 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314}
I20260925 04:41:27.101580 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:27.087929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51358 (local address 127.0.98.62:45827)
0925 04:41:27.088063 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 04:41:27.088068 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:41:27.088683 (+ 615us) server_negotiation.cc:407] Connection header received
0925 04:41:27.089022 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:41:27.089026 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:41:27.089077 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:41:27.089127 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:41:27.090815 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:27.091631 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:27.093758 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:27.094094 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:27.094174 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:27.097119 (+ 2945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:41:27.097124 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:41:27.097126 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:41:27.097159 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:41:27.099320 (+ 2161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:27.099801 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:27.099805 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:27.099807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:27.099903 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:27.100187 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:27.100191 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:27.100192 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:27.100423 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:41:27.100561 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 04:41:27.101306 (+ 745us) server_negotiation.cc:299] Negotiation successful
0925 04:41:27.101426 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 04:41:27.107302 13757 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 229a2e6b7f16481abe3f9bc64f6623ad [term 1 FOLLOWER]: Refusing update from remote peer 1efed41a4dce415180a0a488b19936bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:41:27.108081 13977 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1efed41a4dce415180a0a488b19936bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "229a2e6b7f16481abe3f9bc64f6623ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:41:27.111429 13941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:27.101024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54724 (local address 127.0.98.60:42049)
0925 04:41:27.101168 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 04:41:27.101172 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:41:27.101188 (+ 16us) server_negotiation.cc:407] Connection header received
0925 04:41:27.101230 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:41:27.101233 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:41:27.101278 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:41:27.101351 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:41:27.102179 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:27.102972 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:27.104221 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:27.104576 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:27.104674 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:27.106237 (+ 1563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:41:27.106242 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:41:27.106244 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:41:27.106275 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:41:27.108881 (+ 2606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:27.109538 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:27.109542 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:27.109544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:27.109635 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:27.110116 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:27.110120 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:27.110122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:27.110385 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:41:27.110510 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 04:41:27.110905 (+ 395us) server_negotiation.cc:299] Negotiation successful
0925 04:41:27.111117 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 04:41:27.113360 13945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 229a2e6b7f16481abe3f9bc64f6623ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1efed41a4dce415180a0a488b19936bf. Latest consensus state: current_term: 1 leader_uuid: "1efed41a4dce415180a0a488b19936bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "229a2e6b7f16481abe3f9bc64f6623ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45827 } } peers { permanent_uuid: "1efed41a4dce415180a0a488b19936bf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37711 } } peers { permanent_uuid: "2bf8f8f99fd5489798840d6d1b7f4e0f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42049 } } }
I20260925 04:41:27.113509 13945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 229a2e6b7f16481abe3f9bc64f6623ad [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:41:27.117802 13977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1efed41a4dce415180a0a488b19936bf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1efed41a4dce415180a0a488b19936bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "229a2e6b7f16481abe3f9bc64f6623ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45827 } } peers { permanent_uuid: "1efed41a4dce415180a0a488b19936bf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37711 } } peers { permanent_uuid: "2bf8f8f99fd5489798840d6d1b7f4e0f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42049 } } }
I20260925 04:41:27.117959 13977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1efed41a4dce415180a0a488b19936bf [sys.catalog]: This master's current role is: LEADER
I20260925 04:41:27.119315 13945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 229a2e6b7f16481abe3f9bc64f6623ad [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1efed41a4dce415180a0a488b19936bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "229a2e6b7f16481abe3f9bc64f6623ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45827 } } peers { permanent_uuid: "1efed41a4dce415180a0a488b19936bf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37711 } } peers { permanent_uuid: "2bf8f8f99fd5489798840d6d1b7f4e0f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42049 } } }
I20260925 04:41:27.119468 13945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 229a2e6b7f16481abe3f9bc64f6623ad [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:41:27.119834 13980 catalog_manager.cc:1347] Generated new cluster ID: d54f5c296d704cb0b337cac520820e26
I20260925 04:41:27.119903 13980 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:41:27.125813 13936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bf8f8f99fd5489798840d6d1b7f4e0f: No bootstrap required, opened a new log
I20260925 04:41:27.130590 13936 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2bf8f8f99fd5489798840d6d1b7f4e0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "229a2e6b7f16481abe3f9bc64f6623ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45827 } } peers { permanent_uuid: "1efed41a4dce415180a0a488b19936bf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37711 } } peers { permanent_uuid: "2bf8f8f99fd5489798840d6d1b7f4e0f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42049 } }
I20260925 04:41:27.130872 13936 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2bf8f8f99fd5489798840d6d1b7f4e0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:41:27.130915 13936 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2bf8f8f99fd5489798840d6d1b7f4e0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bf8f8f99fd5489798840d6d1b7f4e0f, State: Initialized, Role: FOLLOWER
I20260925 04:41:27.131393 13936 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2bf8f8f99fd5489798840d6d1b7f4e0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "229a2e6b7f16481abe3f9bc64f6623ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45827 } } peers { permanent_uuid: "1efed41a4dce415180a0a488b19936bf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37711 } } peers { permanent_uuid: "2bf8f8f99fd5489798840d6d1b7f4e0f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42049 } }
I20260925 04:41:27.132167 13986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bf8f8f99fd5489798840d6d1b7f4e0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "229a2e6b7f16481abe3f9bc64f6623ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45827 } } peers { permanent_uuid: "1efed41a4dce415180a0a488b19936bf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37711 } } peers { permanent_uuid: "2bf8f8f99fd5489798840d6d1b7f4e0f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42049 } } }
I20260925 04:41:27.132462 13986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bf8f8f99fd5489798840d6d1b7f4e0f [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:41:27.132887 13936 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2bf8f8f99fd5489798840d6d1b7f4e0f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:41:27.132947 13909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1efed41a4dce415180a0a488b19936bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bf8f8f99fd5489798840d6d1b7f4e0f" is_pre_election: true
I20260925 04:41:27.133250 13909 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2bf8f8f99fd5489798840d6d1b7f4e0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1efed41a4dce415180a0a488b19936bf in term 0.
I20260925 04:41:27.134284 13908 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2bf8f8f99fd5489798840d6d1b7f4e0f [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:41:27.135870 13908 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2bf8f8f99fd5489798840d6d1b7f4e0f [term 1 FOLLOWER]: Refusing update from remote peer 1efed41a4dce415180a0a488b19936bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:41:27.137164 13910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1efed41a4dce415180a0a488b19936bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bf8f8f99fd5489798840d6d1b7f4e0f"
I20260925 04:41:27.138439 13910 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2bf8f8f99fd5489798840d6d1b7f4e0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1efed41a4dce415180a0a488b19936bf in term 1.
I20260925 04:41:27.139842 13946 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1efed41a4dce415180a0a488b19936bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bf8f8f99fd5489798840d6d1b7f4e0f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:41:27.143855 13980 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:41:27.145169 13980 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:41:27.145413 13986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bf8f8f99fd5489798840d6d1b7f4e0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1efed41a4dce415180a0a488b19936bf. Latest consensus state: current_term: 1 leader_uuid: "1efed41a4dce415180a0a488b19936bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "229a2e6b7f16481abe3f9bc64f6623ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45827 } } peers { permanent_uuid: "1efed41a4dce415180a0a488b19936bf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37711 } } peers { permanent_uuid: "2bf8f8f99fd5489798840d6d1b7f4e0f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42049 } } }
I20260925 04:41:27.145546 13986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bf8f8f99fd5489798840d6d1b7f4e0f [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:41:27.145614 13986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bf8f8f99fd5489798840d6d1b7f4e0f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1efed41a4dce415180a0a488b19936bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "229a2e6b7f16481abe3f9bc64f6623ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45827 } } peers { permanent_uuid: "1efed41a4dce415180a0a488b19936bf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37711 } } peers { permanent_uuid: "2bf8f8f99fd5489798840d6d1b7f4e0f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42049 } } }
I20260925 04:41:27.145671 13986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bf8f8f99fd5489798840d6d1b7f4e0f [sys.catalog]: This master's current role is: FOLLOWER
W20260925 04:41:27.149528 13997 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2bf8f8f99fd5489798840d6d1b7f4e0f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 04:41:27.149669 13997 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 04:41:27.154050 13998 mvcc.cc:205] Tried to move back new op lower bound from 7333115031981928448 to 7333115031937826816. Current Snapshot: MvccSnapshot[applied={T|T < 7333115031981928448}]
I20260925 04:41:27.156435 13980 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1efed41a4dce415180a0a488b19936bf: Generated new TSK 0
I20260925 04:41:27.157208 13980 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:41:27.160516 13946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1efed41a4dce415180a0a488b19936bf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1efed41a4dce415180a0a488b19936bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "229a2e6b7f16481abe3f9bc64f6623ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45827 } } peers { permanent_uuid: "1efed41a4dce415180a0a488b19936bf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37711 } } peers { permanent_uuid: "2bf8f8f99fd5489798840d6d1b7f4e0f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42049 } } }
I20260925 04:41:27.160686 13946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1efed41a4dce415180a0a488b19936bf [sys.catalog]: This master's current role is: LEADER
I20260925 04:41:28.071137 13959 catalog_manager.cc:1259] Loaded cluster ID: d54f5c296d704cb0b337cac520820e26
I20260925 04:41:28.071246 13959 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 229a2e6b7f16481abe3f9bc64f6623ad: loading cluster ID for follower catalog manager: success
I20260925 04:41:28.073086 13959 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 229a2e6b7f16481abe3f9bc64f6623ad: acquiring CA information for follower catalog manager: success
I20260925 04:41:28.074434 13959 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 229a2e6b7f16481abe3f9bc64f6623ad: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 04:41:28.151087 13997 catalog_manager.cc:1259] Loaded cluster ID: d54f5c296d704cb0b337cac520820e26
I20260925 04:41:28.151181 13997 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2bf8f8f99fd5489798840d6d1b7f4e0f: loading cluster ID for follower catalog manager: success
I20260925 04:41:28.152912 13997 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2bf8f8f99fd5489798840d6d1b7f4e0f: acquiring CA information for follower catalog manager: success
I20260925 04:41:28.154269 13997 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2bf8f8f99fd5489798840d6d1b7f4e0f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 04:41:28.225195 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 13715
I20260925 04:41:28.231017 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 13790
I20260925 04:41:28.236596 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 13865
2026-09-25T04:41:28Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2997 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790311123478858-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:41:28 dist-test-slave-dc95 krb5kdc[14010](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:41:28 dist-test-slave-dc95 krb5kdc[14010](info): set up 2 sockets
Sep 25 04:41:28 dist-test-slave-dc95 krb5kdc[14010](info): commencing operation
krb5kdc: starting...
W20260925 04:41:29.195592 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790311123478858-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790311123478858-392-0/krb5kdc/test-user.keytab.
Sep 25 04:41:29 dist-test-slave-dc95 krb5kdc[14010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:41:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:41:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 04:41:29.318949 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790311123478858-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790311123478858-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790311123478858-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44839
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44839
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:41:29.432417 14023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:41:29.432734 14023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:41:29.432776 14023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:41:29.437254 14023 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:41:29.437350 14023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:41:29.437371 14023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:41:29.437388 14023 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:41:29.437402 14023 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:41:29.443215 14023 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790311123478858-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790311123478858-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790311123478858-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:41:29.444403 14023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:41:29.445420 14023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:41:29.454052 14029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:41:29.454046 14028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:41:29.454169 14023 server_base.cc:1034] running on GCE node
W20260925 04:41:29.454018 14031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:41:29.454916 14023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:41:29.456136 14023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:41:29.457326 14023 hybrid_clock.cc:648] HybridClock initialized: now 1790311289457310 us; error 31 us; skew 500 ppm
Sep 25 04:41:29 dist-test-slave-dc95 krb5kdc[14010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311289, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:41:29.460296 14023 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 04:41:29.461268 14023 webserver.cc:460] Webserver started at http://127.0.98.62:37331/ using document root <none> and password file <none>
I20260925 04:41:29.461822 14023 fs_manager.cc:359] Metadata directory not provided
I20260925 04:41:29.461984 14023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:41:29.462280 14023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:41:29.464486 14023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790311123478858-392-0/minicluster-data/master-0/data/instance:
uuid: "61eda7423ddf4893933d56d8b46aafb2"
format_stamp: "Formatted at 2026-09-25 04:41:29 on dist-test-slave-dc95"
server_key: "96c990fb9294c7adc80b07db3ce304fd"
server_key_iv: "9071bd60ba188bb4856beb645fc51a28"
server_key_version: "encryptionkey@0"
I20260925 04:41:29.465040 14023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790311123478858-392-0/minicluster-data/master-0/wal/instance:
uuid: "61eda7423ddf4893933d56d8b46aafb2"
format_stamp: "Formatted at 2026-09-25 04:41:29 on dist-test-slave-dc95"
server_key: "96c990fb9294c7adc80b07db3ce304fd"
server_key_iv: "9071bd60ba188bb4856beb645fc51a28"
server_key_version: "encryptionkey@0"
I20260925 04:41:29.469094 14023 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:41:29.471824 14038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:41:29.473243 14023 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:41:29.473384 14023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790311123478858-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790311123478858-392-0/minicluster-data/master-0/wal
uuid: "61eda7423ddf4893933d56d8b46aafb2"
format_stamp: "Formatted at 2026-09-25 04:41:29 on dist-test-slave-dc95"
server_key: "96c990fb9294c7adc80b07db3ce304fd"
server_key_iv: "9071bd60ba188bb4856beb645fc51a28"
server_key_version: "encryptionkey@0"
I20260925 04:41:29.473508 14023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790311123478858-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790311123478858-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790311123478858-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:41:29.485015 14023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:41:29.488889 14023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:41:29.489095 14023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:41:29.496292 14023 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44839
I20260925 04:41:29.496361 14090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44839 every 8 connection(s)
I20260925 04:41:29.497280 14023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
I20260925 04:41:29.500383 14091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:41:29.505640 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 14023
I20260925 04:41:29.505785 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790311123478858-392-0/minicluster-data/master-0/wal/instance
I20260925 04:41:29.506068 392 external_mini_cluster.cc:1428] Setting key bce3bad1b8beed87e2212df116c92ed7
I20260925 04:41:29.507494 14091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61eda7423ddf4893933d56d8b46aafb2: Bootstrap starting.
I20260925 04:41:29.510483 14091 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61eda7423ddf4893933d56d8b46aafb2: Neither blocks nor log segments found. Creating new log.
I20260925 04:41:29.511358 14091 log.cc:824] T 00000000000000000000000000000000 P 61eda7423ddf4893933d56d8b46aafb2: Log is configured to *not* fsync() on all Append() calls
I20260925 04:41:29.513970 14091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61eda7423ddf4893933d56d8b46aafb2: No bootstrap required, opened a new log
Sep 25 04:41:29 dist-test-slave-dc95 krb5kdc[14010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:41:29.518213 14091 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61eda7423ddf4893933d56d8b46aafb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61eda7423ddf4893933d56d8b46aafb2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44839 } }
I20260925 04:41:29.518525 14091 raft_consensus.cc:374] T 00000000000000000000000000000000 P 61eda7423ddf4893933d56d8b46aafb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:41:29.518580 14091 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61eda7423ddf4893933d56d8b46aafb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61eda7423ddf4893933d56d8b46aafb2, State: Initialized, Role: FOLLOWER
I20260925 04:41:29.519114 14091 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61eda7423ddf4893933d56d8b46aafb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61eda7423ddf4893933d56d8b46aafb2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44839 } }
I20260925 04:41:29.519265 14091 raft_consensus.cc:388] T 00000000000000000000000000000000 P 61eda7423ddf4893933d56d8b46aafb2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:41:29.519338 14091 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61eda7423ddf4893933d56d8b46aafb2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:41:29.519439 14091 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61eda7423ddf4893933d56d8b46aafb2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:41:29.520605 14091 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61eda7423ddf4893933d56d8b46aafb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61eda7423ddf4893933d56d8b46aafb2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44839 } }
I20260925 04:41:29.520979 14091 leader_election.cc:302] T 00000000000000000000000000000000 P 61eda7423ddf4893933d56d8b46aafb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61eda7423ddf4893933d56d8b46aafb2; no voters:
I20260925 04:41:29.521308 14091 leader_election.cc:288] T 00000000000000000000000000000000 P 61eda7423ddf4893933d56d8b46aafb2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:41:29.521410 14096 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61eda7423ddf4893933d56d8b46aafb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:41:29.521613 14096 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61eda7423ddf4893933d56d8b46aafb2 [term 1 LEADER]: Becoming Leader. State: Replica: 61eda7423ddf4893933d56d8b46aafb2, State: Running, Role: LEADER
I20260925 04:41:29.521942 14096 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61eda7423ddf4893933d56d8b46aafb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61eda7423ddf4893933d56d8b46aafb2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44839 } }
I20260925 04:41:29.522746 14091 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61eda7423ddf4893933d56d8b46aafb2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:41:29.523480 14094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:29.507718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38150 (local address 127.0.98.62:44839)
0925 04:41:29.508158 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0925 04:41:29.508178 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 04:41:29.508218 (+ 40us) server_negotiation.cc:407] Connection header received
0925 04:41:29.509136 (+ 918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:41:29.509163 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:41:29.509611 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:41:29.510012 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:41:29.511070 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:29.512447 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:29.513734 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:29.514454 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:29.514638 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:29.517722 (+ 3084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:41:29.517728 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:41:29.517745 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:41:29.517778 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:41:29.519877 (+ 2099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:29.520507 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:29.520516 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:29.520522 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:29.520660 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:29.520979 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:29.520982 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:29.520984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:29.521514 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:41:29.521710 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0925 04:41:29.522186 (+ 476us) server_negotiation.cc:299] Negotiation successful
0925 04:41:29.522453 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":124,"threads_started":1}
I20260925 04:41:29.524266 14098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61eda7423ddf4893933d56d8b46aafb2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "61eda7423ddf4893933d56d8b46aafb2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61eda7423ddf4893933d56d8b46aafb2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44839 } } }
I20260925 04:41:29.524472 14098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61eda7423ddf4893933d56d8b46aafb2 [sys.catalog]: This master's current role is: LEADER
I20260925 04:41:29.524904 14097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61eda7423ddf4893933d56d8b46aafb2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61eda7423ddf4893933d56d8b46aafb2. Latest consensus state: current_term: 1 leader_uuid: "61eda7423ddf4893933d56d8b46aafb2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61eda7423ddf4893933d56d8b46aafb2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44839 } } }
I20260925 04:41:29.525051 14097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61eda7423ddf4893933d56d8b46aafb2 [sys.catalog]: This master's current role is: LEADER
I20260925 04:41:29.528258 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260925 04:41:29.529045 14111 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 61eda7423ddf4893933d56d8b46aafb2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 04:41:29.529148 14111 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 04:41:29.529224 14105 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:41:29.529933 14105 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:41:29.536852 14105 catalog_manager.cc:1347] Generated new cluster ID: e03e3419b6974a67bb6a794b35e98cc9
I20260925 04:41:29.536943 14105 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:41:29.548708 14105 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:41:29.549844 14105 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:41:29.562005 14105 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 61eda7423ddf4893933d56d8b46aafb2: Generated new TSK 0
I20260925 04:41:29.562767 14105 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:41:29.584957 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 14023
2026-09-25T04:41:29Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1346 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:41:29 dist-test-slave-dc95 krb5kdc[14118](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:41:29 dist-test-slave-dc95 krb5kdc[14118](info): set up 2 sockets
Sep 25 04:41:29 dist-test-slave-dc95 krb5kdc[14118](info): commencing operation
krb5kdc: starting...
W20260925 04:41:30.543888 392 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/krb5kdc/test-user.keytab.
Sep 25 04:41:30 dist-test-slave-dc95 krb5kdc[14118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:41:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:41:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 04:41:30.671221 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46403
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46403
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:41:30.784214 14131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:41:30.784521 14131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:41:30.784564 14131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:41:30.788806 14131 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:41:30.788892 14131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:41:30.788913 14131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:41:30.788929 14131 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:41:30.788944 14131 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:41:30.795140 14131 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46403
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14131
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:41:30.796389 14131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:41:30.797430 14131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:41:30.806861 14137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:41:30.807027 14131 server_base.cc:1034] running on GCE node
W20260925 04:41:30.806768 14139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:41:30.806861 14136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:41:30.807823 14131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:41:30.809056 14131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:41:30.810276 14131 hybrid_clock.cc:648] HybridClock initialized: now 1790311290810243 us; error 53 us; skew 500 ppm
Sep 25 04:41:30 dist-test-slave-dc95 krb5kdc[14118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311290, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:41:30.819280 14131 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 04:41:30.820421 14131 webserver.cc:460] Webserver started at http://127.0.98.62:34573/ using document root <none> and password file <none>
I20260925 04:41:30.821022 14131 fs_manager.cc:359] Metadata directory not provided
I20260925 04:41:30.821094 14131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:41:30.821305 14131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:41:30.823750 14131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/master-0/data/instance:
uuid: "5ad9c8d894c349b58d60ea2e39f3d704"
format_stamp: "Formatted at 2026-09-25 04:41:30 on dist-test-slave-dc95"
server_key: "1a99b4b373f3cbe241fa387b7cee2418"
server_key_iv: "a44e137621847188cdc1c3967ae690c3"
server_key_version: "encryptionkey@0"
I20260925 04:41:30.824646 14131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/master-0/wal/instance:
uuid: "5ad9c8d894c349b58d60ea2e39f3d704"
format_stamp: "Formatted at 2026-09-25 04:41:30 on dist-test-slave-dc95"
server_key: "1a99b4b373f3cbe241fa387b7cee2418"
server_key_iv: "a44e137621847188cdc1c3967ae690c3"
server_key_version: "encryptionkey@0"
I20260925 04:41:30.829226 14131 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:41:30.832264 14146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:41:30.833916 14131 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:41:30.834074 14131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/master-0/wal
uuid: "5ad9c8d894c349b58d60ea2e39f3d704"
format_stamp: "Formatted at 2026-09-25 04:41:30 on dist-test-slave-dc95"
server_key: "1a99b4b373f3cbe241fa387b7cee2418"
server_key_iv: "a44e137621847188cdc1c3967ae690c3"
server_key_version: "encryptionkey@0"
I20260925 04:41:30.834233 14131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:41:30.862103 14131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:41:30.865666 14131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:41:30.865862 14131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:41:30.873646 14131 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46403
I20260925 04:41:30.873917 14198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46403 every 8 connection(s)
I20260925 04:41:30.874675 14131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
I20260925 04:41:30.877734 14199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:41:30.878293 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 14131
I20260925 04:41:30.878414 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/master-0/wal/instance
I20260925 04:41:30.878743 392 external_mini_cluster.cc:1428] Setting key 30b39e9959d9e1c86bd0125156c40e32
I20260925 04:41:30.884883 14199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ad9c8d894c349b58d60ea2e39f3d704: Bootstrap starting.
I20260925 04:41:30.887858 14199 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ad9c8d894c349b58d60ea2e39f3d704: Neither blocks nor log segments found. Creating new log.
Sep 25 04:41:30 dist-test-slave-dc95 krb5kdc[14118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:41:30.888715 14199 log.cc:824] T 00000000000000000000000000000000 P 5ad9c8d894c349b58d60ea2e39f3d704: Log is configured to *not* fsync() on all Append() calls
I20260925 04:41:30.891273 14199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ad9c8d894c349b58d60ea2e39f3d704: No bootstrap required, opened a new log
I20260925 04:41:30.895278 14202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:30.880498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57310 (local address 127.0.98.62:46403)
0925 04:41:30.881149 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0925 04:41:30.881172 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0925 04:41:30.881222 (+ 50us) server_negotiation.cc:407] Connection header received
0925 04:41:30.881986 (+ 764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:41:30.882007 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:41:30.882386 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:41:30.882691 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:41:30.883584 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:30.884680 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:30.885842 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:30.886344 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:30.886476 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:30.889327 (+ 2851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:41:30.889335 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:41:30.889355 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:41:30.889385 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:41:30.891413 (+ 2028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:30.892209 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:30.892220 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:30.892228 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:30.892597 (+ 369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:30.893010 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:30.893014 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:30.893016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:30.893523 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:41:30.893704 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0925 04:41:30.894094 (+ 390us) server_negotiation.cc:299] Negotiation successful
0925 04:41:30.894430 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":145,"threads_started":1}
I20260925 04:41:30.895473 14199 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ad9c8d894c349b58d60ea2e39f3d704 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad9c8d894c349b58d60ea2e39f3d704" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46403 } }
I20260925 04:41:30.895817 14199 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ad9c8d894c349b58d60ea2e39f3d704 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:41:30.895908 14199 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ad9c8d894c349b58d60ea2e39f3d704 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ad9c8d894c349b58d60ea2e39f3d704, State: Initialized, Role: FOLLOWER
I20260925 04:41:30.896358 14199 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ad9c8d894c349b58d60ea2e39f3d704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad9c8d894c349b58d60ea2e39f3d704" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46403 } }
I20260925 04:41:30.896495 14199 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ad9c8d894c349b58d60ea2e39f3d704 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:41:30.896559 14199 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ad9c8d894c349b58d60ea2e39f3d704 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:41:30.896658 14199 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ad9c8d894c349b58d60ea2e39f3d704 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:41:30.897653 14199 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ad9c8d894c349b58d60ea2e39f3d704 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad9c8d894c349b58d60ea2e39f3d704" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46403 } }
I20260925 04:41:30.897991 14199 leader_election.cc:302] T 00000000000000000000000000000000 P 5ad9c8d894c349b58d60ea2e39f3d704 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ad9c8d894c349b58d60ea2e39f3d704; no voters:
I20260925 04:41:30.898306 14199 leader_election.cc:288] T 00000000000000000000000000000000 P 5ad9c8d894c349b58d60ea2e39f3d704 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:41:30.898435 14204 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ad9c8d894c349b58d60ea2e39f3d704 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:41:30.898648 14204 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ad9c8d894c349b58d60ea2e39f3d704 [term 1 LEADER]: Becoming Leader. State: Replica: 5ad9c8d894c349b58d60ea2e39f3d704, State: Running, Role: LEADER
I20260925 04:41:30.898984 14204 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ad9c8d894c349b58d60ea2e39f3d704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad9c8d894c349b58d60ea2e39f3d704" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46403 } }
I20260925 04:41:30.900463 14199 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ad9c8d894c349b58d60ea2e39f3d704 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:41:30.900435 14205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ad9c8d894c349b58d60ea2e39f3d704 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ad9c8d894c349b58d60ea2e39f3d704. Latest consensus state: current_term: 1 leader_uuid: "5ad9c8d894c349b58d60ea2e39f3d704" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad9c8d894c349b58d60ea2e39f3d704" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46403 } } }
I20260925 04:41:30.900648 14205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ad9c8d894c349b58d60ea2e39f3d704 [sys.catalog]: This master's current role is: LEADER
I20260925 04:41:30.900806 14206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ad9c8d894c349b58d60ea2e39f3d704 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ad9c8d894c349b58d60ea2e39f3d704" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad9c8d894c349b58d60ea2e39f3d704" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46403 } } }
I20260925 04:41:30.900971 14206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ad9c8d894c349b58d60ea2e39f3d704 [sys.catalog]: This master's current role is: LEADER
I20260925 04:41:30.901119 14208 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:41:30.904443 14208 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:41:30.910727 14208 catalog_manager.cc:1347] Generated new cluster ID: 1235377a389c4d639f6e15e42f920f1f
I20260925 04:41:30.910820 14208 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:41:30.941996 14208 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:41:30.943017 14208 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:41:30.957319 14208 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ad9c8d894c349b58d60ea2e39f3d704: Generated new TSK 0
I20260925 04:41:30.958091 14208 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 04:41:30.991581 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46403
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:41:31.111488 14225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:41:31.111842 14225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:41:31.111904 14225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:41:31.116175 14225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:41:31.116262 14225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:41:31.116348 14225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 04:41:31.121979 14225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14225
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:41:31.123126 14225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:41:31.124209 14225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:41:31.134161 14230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:41:31.134172 14233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:41:31.134161 14231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:41:31.134377 14225 server_base.cc:1034] running on GCE node
I20260925 04:41:31.135016 14225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:41:31.135776 14225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:41:31.136967 14225 hybrid_clock.cc:648] HybridClock initialized: now 1790311291136948 us; error 35 us; skew 500 ppm
Sep 25 04:41:31 dist-test-slave-dc95 krb5kdc[14118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311291, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:41:31.140019 14225 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 04:41:31.140974 14225 webserver.cc:460] Webserver started at http://127.0.98.1:35049/ using document root <none> and password file <none>
I20260925 04:41:31.141542 14225 fs_manager.cc:359] Metadata directory not provided
I20260925 04:41:31.141610 14225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:41:31.141813 14225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:41:31.143965 14225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-0/data/instance:
uuid: "7a781f32e0144cbf99f8bbff69c3fdac"
format_stamp: "Formatted at 2026-09-25 04:41:31 on dist-test-slave-dc95"
server_key: "6893d856fc402642507b48120ed8ef4a"
server_key_iv: "3424d3d43039b8b6596f7353b3cd293a"
server_key_version: "encryptionkey@0"
I20260925 04:41:31.144528 14225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7a781f32e0144cbf99f8bbff69c3fdac"
format_stamp: "Formatted at 2026-09-25 04:41:31 on dist-test-slave-dc95"
server_key: "6893d856fc402642507b48120ed8ef4a"
server_key_iv: "3424d3d43039b8b6596f7353b3cd293a"
server_key_version: "encryptionkey@0"
I20260925 04:41:31.148599 14225 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 04:41:31.151472 14240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:41:31.153012 14225 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 04:41:31.153163 14225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-0/wal
uuid: "7a781f32e0144cbf99f8bbff69c3fdac"
format_stamp: "Formatted at 2026-09-25 04:41:31 on dist-test-slave-dc95"
server_key: "6893d856fc402642507b48120ed8ef4a"
server_key_iv: "3424d3d43039b8b6596f7353b3cd293a"
server_key_version: "encryptionkey@0"
I20260925 04:41:31.153306 14225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:41:31.171818 14225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:41:31.175282 14225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:41:31.175503 14225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:41:31.176178 14225 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:41:31.177065 14225 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:41:31.177112 14225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:41:31.177155 14225 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:41:31.177182 14225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:41:31.185940 14225 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34513
I20260925 04:41:31.185974 14353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34513 every 8 connection(s)
I20260925 04:41:31.187024 14225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
I20260925 04:41:31.188725 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 14225
I20260925 04:41:31.188846 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-0/wal/instance
I20260925 04:41:31.189134 392 external_mini_cluster.cc:1428] Setting key 42b9f27cd66a0c687a51623824f2c560
Sep 25 04:41:31 dist-test-slave-dc95 krb5kdc[14118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311291, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260925 04:41:31.203203 14202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:31.190568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56639 (local address 127.0.98.62:46403)
0925 04:41:31.190703 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 04:41:31.190708 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:41:31.190722 (+ 14us) server_negotiation.cc:407] Connection header received
0925 04:41:31.191703 (+ 981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:41:31.191707 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:41:31.191762 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:41:31.191850 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:41:31.193386 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:31.194177 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:31.195449 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:31.195865 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:31.195986 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:31.198979 (+ 2993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:41:31.198985 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:41:31.198988 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:41:31.199023 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:41:31.200635 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:31.201320 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:31.201323 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:31.201324 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:31.201421 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:31.201837 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:31.201840 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:31.201842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:31.202082 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:41:31.202217 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 04:41:31.202919 (+ 702us) server_negotiation.cc:299] Negotiation successful
0925 04:41:31.203070 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 04:41:31.204288 14356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:31.189538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.1:56639)
0925 04:41:31.190535 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:31.190587 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 04:41:31.191465 (+ 878us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:41:31.192121 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:41:31.192130 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:41:31.192608 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:41:31.193196 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:31.193212 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:31.194313 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:31.194315 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:31.195306 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:31.195318 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:31.196045 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:31.196049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:31.196224 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:31.196940 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:41:31.196960 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:41:31.198713 (+ 1753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:41:31.200827 (+ 2114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:41:31.200834 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:41:31.200850 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:41:31.201218 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:41:31.201534 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:41:31.201538 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:41:31.201539 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:41:31.201697 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:41:31.202220 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:41:31.202228 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:41:31.202600 (+ 372us) client_negotiation.cc:769] Sending connection context
0925 04:41:31.202830 (+ 230us) client_negotiation.cc:240] Negotiation successful
0925 04:41:31.203203 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":753,"thread_start_us":157,"threads_started":1}
I20260925 04:41:31.205619 14354 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46403
I20260925 04:41:31.205870 14354 heartbeater.cc:464] Registering TS with master...
I20260925 04:41:31.206434 14354 heartbeater.cc:511] Master 127.0.98.62:46403 requested a full tablet report, sending...
I20260925 04:41:31.208158 14162 ts_manager.cc:194] Registered new tserver with Master: 7a781f32e0144cbf99f8bbff69c3fdac (127.0.98.1:34513)
I20260925 04:41:31.209318 14162 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56639
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 04:41:31.219415 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46403
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:41:31.335070 14359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:41:31.335376 14359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:41:31.335441 14359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:41:31.339814 14359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:41:31.339901 14359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:41:31.339982 14359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 04:41:31.345767 14359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:41:31.346935 14359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:41:31.348110 14359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:41:31.358433 14364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:41:31.358526 14365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:41:31.358440 14367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:41:31.358803 14359 server_base.cc:1034] running on GCE node
I20260925 04:41:31.359284 14359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:41:31.360073 14359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:41:31.361258 14359 hybrid_clock.cc:648] HybridClock initialized: now 1790311291361234 us; error 39 us; skew 500 ppm
Sep 25 04:41:31 dist-test-slave-dc95 krb5kdc[14118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311291, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:41:31.364229 14359 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 04:41:31.365262 14359 webserver.cc:460] Webserver started at http://127.0.98.2:36097/ using document root <none> and password file <none>
I20260925 04:41:31.365871 14359 fs_manager.cc:359] Metadata directory not provided
I20260925 04:41:31.365937 14359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:41:31.366111 14359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:41:31.368351 14359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-1/data/instance:
uuid: "08cb310fd578413c950d87af40935374"
format_stamp: "Formatted at 2026-09-25 04:41:31 on dist-test-slave-dc95"
server_key: "10bb2b5d75ba0c0ee3f56bdfe9fa35c0"
server_key_iv: "51cc0695b753f9e04434fb1294aa76f6"
server_key_version: "encryptionkey@0"
I20260925 04:41:31.368925 14359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-1/wal/instance:
uuid: "08cb310fd578413c950d87af40935374"
format_stamp: "Formatted at 2026-09-25 04:41:31 on dist-test-slave-dc95"
server_key: "10bb2b5d75ba0c0ee3f56bdfe9fa35c0"
server_key_iv: "51cc0695b753f9e04434fb1294aa76f6"
server_key_version: "encryptionkey@0"
I20260925 04:41:31.372905 14359 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260925 04:41:31.375525 14374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:41:31.376991 14359 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 04:41:31.377127 14359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-1/wal
uuid: "08cb310fd578413c950d87af40935374"
format_stamp: "Formatted at 2026-09-25 04:41:31 on dist-test-slave-dc95"
server_key: "10bb2b5d75ba0c0ee3f56bdfe9fa35c0"
server_key_iv: "51cc0695b753f9e04434fb1294aa76f6"
server_key_version: "encryptionkey@0"
I20260925 04:41:31.377260 14359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:41:31.419970 14359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:41:31.423439 14359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:41:31.423668 14359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:41:31.424276 14359 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:41:31.425120 14359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:41:31.425166 14359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:41:31.425208 14359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:41:31.425237 14359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:41:31.433861 14359 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41513
I20260925 04:41:31.433915 14487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41513 every 8 connection(s)
I20260925 04:41:31.434772 14359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
I20260925 04:41:31.436597 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 14359
I20260925 04:41:31.436730 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-1/wal/instance
I20260925 04:41:31.437031 392 external_mini_cluster.cc:1428] Setting key 3a9101775f902624c9df41f5c3d01fea
Sep 25 04:41:31 dist-test-slave-dc95 krb5kdc[14118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311291, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260925 04:41:31.451028 14202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:31.436812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45845 (local address 127.0.98.62:46403)
0925 04:41:31.436981 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 04:41:31.436985 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:41:31.438016 (+ 1031us) server_negotiation.cc:407] Connection header received
0925 04:41:31.439783 (+ 1767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:41:31.439786 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:41:31.439836 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:41:31.439964 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:41:31.442437 (+ 2473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:31.442995 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:31.444306 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:31.444539 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:31.444633 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:31.447340 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:41:31.447343 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:41:31.447345 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:41:31.447373 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:41:31.448690 (+ 1317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:31.449321 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:31.449325 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:31.449326 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:31.449418 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:31.449792 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:31.449795 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:31.449797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:31.450020 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:41:31.450126 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 04:41:31.450708 (+ 582us) server_negotiation.cc:299] Negotiation successful
0925 04:41:31.450859 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 04:41:31.452011 14490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:31.437331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.2:45845)
0925 04:41:31.437857 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:31.437915 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 04:41:31.439496 (+ 1581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:41:31.440850 (+ 1354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:41:31.440861 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:41:31.441391 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:41:31.442205 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:31.442222 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:31.443134 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:31.443138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:31.444170 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:31.444183 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:31.444702 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:31.444706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:31.444879 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:31.445553 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:41:31.445571 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:41:31.447130 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:41:31.448819 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:41:31.448827 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:41:31.448846 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:41:31.449212 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:41:31.449527 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:41:31.449530 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:41:31.449532 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:41:31.449698 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:41:31.450159 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:41:31.450174 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:41:31.450446 (+ 272us) client_negotiation.cc:769] Sending connection context
0925 04:41:31.450627 (+ 181us) client_negotiation.cc:240] Negotiation successful
0925 04:41:31.450943 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":121,"threads_started":1}
I20260925 04:41:31.453411 14488 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46403
I20260925 04:41:31.453688 14488 heartbeater.cc:464] Registering TS with master...
I20260925 04:41:31.454272 14488 heartbeater.cc:511] Master 127.0.98.62:46403 requested a full tablet report, sending...
I20260925 04:41:31.455556 14162 ts_manager.cc:194] Registered new tserver with Master: 08cb310fd578413c950d87af40935374 (127.0.98.2:41513)
I20260925 04:41:31.456398 14162 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45845
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 04:41:31.468024 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46403
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:41:31.594784 14493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:41:31.595096 14493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:41:31.595156 14493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:41:31.599462 14493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:41:31.599550 14493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:41:31.599656 14493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 04:41:31.605484 14493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:41:31.606544 14493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:41:31.607553 14493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:41:31.617668 14499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:41:31.617666 14501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:41:31.617723 14493 server_base.cc:1034] running on GCE node
W20260925 04:41:31.617637 14498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:41:31.618341 14493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:41:31.619161 14493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:41:31.620345 14493 hybrid_clock.cc:648] HybridClock initialized: now 1790311291620317 us; error 52 us; skew 500 ppm
Sep 25 04:41:31 dist-test-slave-dc95 krb5kdc[14118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311291, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:41:31.623102 14493 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 04:41:31.624143 14493 webserver.cc:460] Webserver started at http://127.0.98.3:46641/ using document root <none> and password file <none>
I20260925 04:41:31.624756 14493 fs_manager.cc:359] Metadata directory not provided
I20260925 04:41:31.624822 14493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:41:31.625008 14493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:41:31.627187 14493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-2/data/instance:
uuid: "fceaa38dcc7f4a8fae17e2252f431b15"
format_stamp: "Formatted at 2026-09-25 04:41:31 on dist-test-slave-dc95"
server_key: "7d91947786c8621a47da59630897f7c3"
server_key_iv: "d0ec3fc0241e0ade903599fcb32e2cdc"
server_key_version: "encryptionkey@0"
I20260925 04:41:31.627903 14493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-2/wal/instance:
uuid: "fceaa38dcc7f4a8fae17e2252f431b15"
format_stamp: "Formatted at 2026-09-25 04:41:31 on dist-test-slave-dc95"
server_key: "7d91947786c8621a47da59630897f7c3"
server_key_iv: "d0ec3fc0241e0ade903599fcb32e2cdc"
server_key_version: "encryptionkey@0"
I20260925 04:41:31.632009 14493 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:41:31.634582 14508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:41:31.635978 14493 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:41:31.636091 14493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-2/wal
uuid: "fceaa38dcc7f4a8fae17e2252f431b15"
format_stamp: "Formatted at 2026-09-25 04:41:31 on dist-test-slave-dc95"
server_key: "7d91947786c8621a47da59630897f7c3"
server_key_iv: "d0ec3fc0241e0ade903599fcb32e2cdc"
server_key_version: "encryptionkey@0"
I20260925 04:41:31.636207 14493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:41:31.648164 14493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:41:31.651257 14493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:41:31.651453 14493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:41:31.652119 14493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:41:31.652985 14493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:41:31.653031 14493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:41:31.653071 14493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:41:31.653100 14493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:41:31.661953 14493 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44101
I20260925 04:41:31.661993 14621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44101 every 8 connection(s)
I20260925 04:41:31.662878 14493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
I20260925 04:41:31.664880 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 14493
I20260925 04:41:31.665010 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-2/wal/instance
I20260925 04:41:31.665278 392 external_mini_cluster.cc:1428] Setting key 57bbbe5dace248306df0734922bddde9
Sep 25 04:41:31 dist-test-slave-dc95 krb5kdc[14118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311291, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:41:31.676851 14202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:31.664835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51295 (local address 127.0.98.62:46403)
0925 04:41:31.664994 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 04:41:31.664999 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:41:31.665940 (+ 941us) server_negotiation.cc:407] Connection header received
0925 04:41:31.667478 (+ 1538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:41:31.667481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:41:31.667540 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:41:31.667636 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:41:31.668933 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:31.669466 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:31.670653 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:31.670887 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:31.670985 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:31.673509 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:41:31.673513 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:41:31.673515 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:41:31.673540 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:41:31.674818 (+ 1278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:31.675341 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:31.675344 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:31.675345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:31.675421 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:31.675821 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:31.675824 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:31.675825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:31.676017 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:41:31.676094 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 04:41:31.676607 (+ 513us) server_negotiation.cc:299] Negotiation successful
0925 04:41:31.676718 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 04:41:31.677707 14624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:31.665251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.3:51295)
0925 04:41:31.665762 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:31.665824 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0925 04:41:31.667247 (+ 1423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:41:31.667826 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:41:31.667833 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:41:31.668213 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:41:31.668756 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:31.668771 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:31.669604 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:31.669608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:31.670524 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:31.670536 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:31.671050 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:31.671054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:31.671223 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:31.671837 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:41:31.671853 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:41:31.673313 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:41:31.674940 (+ 1627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:41:31.674952 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:41:31.674966 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:41:31.675249 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:41:31.675532 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:41:31.675534 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:41:31.675536 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:41:31.675737 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:41:31.676108 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:41:31.676113 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:41:31.676369 (+ 256us) client_negotiation.cc:769] Sending connection context
0925 04:41:31.676531 (+ 162us) client_negotiation.cc:240] Negotiation successful
0925 04:41:31.676805 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":137,"threads_started":1}
I20260925 04:41:31.678831 14622 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46403
I20260925 04:41:31.679122 14622 heartbeater.cc:464] Registering TS with master...
I20260925 04:41:31.679737 14622 heartbeater.cc:511] Master 127.0.98.62:46403 requested a full tablet report, sending...
I20260925 04:41:31.680814 14162 ts_manager.cc:194] Registered new tserver with Master: fceaa38dcc7f4a8fae17e2252f431b15 (127.0.98.3:44101)
I20260925 04:41:31.681384 14162 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51295
I20260925 04:41:31.691568 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:41:31.701112 14202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:31.693610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57320 (local address 127.0.98.62:46403)
0925 04:41:31.693734 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 04:41:31.693738 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:41:31.694005 (+ 267us) server_negotiation.cc:407] Connection header received
0925 04:41:31.694094 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:41:31.694097 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:41:31.694151 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:41:31.694220 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:41:31.695034 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:31.695551 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:31.696356 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:31.696570 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:31.696647 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:31.698093 (+ 1446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:41:31.698096 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:41:31.698098 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:41:31.698124 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:41:31.699345 (+ 1221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:31.699827 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:31.699830 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:31.699831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:31.699928 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:31.700195 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:31.700197 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:31.700198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:31.700426 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:41:31.700529 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 04:41:31.700874 (+ 345us) server_negotiation.cc:299] Negotiation successful
0925 04:41:31.700982 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 04:41:31.704058 14162 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57320:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:41:31.707167 14162 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:41:31.722397 14634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:31.716040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44101 (local address 127.0.0.1:40446)
0925 04:41:31.716887 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:31.716939 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 04:41:31.717144 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:41:31.717894 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:41:31.717900 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:41:31.717923 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:41:31.718064 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:31.718072 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:31.719584 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:31.719587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:31.720666 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:31.720678 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:31.721873 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:31.721876 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:31.721961 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:31.721991 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 04:41:31.722090 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 04:41:31.722231 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"mutex_wait_us":67,"thread_start_us":74,"threads_started":1}
I20260925 04:41:31.722397 14635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:31.716159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40446 (local address 127.0.98.3:44101)
0925 04:41:31.716638 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0925 04:41:31.716648 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 04:41:31.717032 (+ 384us) server_negotiation.cc:407] Connection header received
0925 04:41:31.717301 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:41:31.717307 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:41:31.717527 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:41:31.717674 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:41:31.718190 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:31.719432 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:31.720801 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:31.721546 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:31.721650 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:31.721815 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 04:41:31.722141 (+ 326us) server_negotiation.cc:299] Negotiation successful
0925 04:41:31.722245 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":119,"threads_started":1}
I20260925 04:41:31.722879 14633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:31.716129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41513 (local address 127.0.0.1:50490)
0925 04:41:31.717356 (+ 1227us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:31.717378 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 04:41:31.717451 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:41:31.717930 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:41:31.717933 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:41:31.717967 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:41:31.718082 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:31.718088 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:31.719560 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:31.719562 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:31.721206 (+ 1644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:31.721218 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:31.722453 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:31.722457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:31.722555 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:31.722577 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 04:41:31.722645 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 04:41:31.722722 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1169,"mutex_wait_us":221,"spinlock_wait_cycles":9856,"thread_start_us":89,"threads_started":1}
I20260925 04:41:31.722949 14636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:31.716877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50490 (local address 127.0.98.2:41513)
0925 04:41:31.717187 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0925 04:41:31.717198 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 04:41:31.717532 (+ 334us) server_negotiation.cc:407] Connection header received
0925 04:41:31.717613 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:41:31.717618 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:41:31.717793 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:41:31.717938 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:41:31.718264 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:31.719433 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:31.721352 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:31.722145 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:31.722249 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:31.722388 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 04:41:31.722701 (+ 313us) server_negotiation.cc:299] Negotiation successful
0925 04:41:31.722798 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":111,"threads_started":1}
I20260925 04:41:31.723018 14637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:31.717143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34513 (local address 127.0.0.1:45092)
0925 04:41:31.717488 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:31.717503 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:41:31.717593 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:41:31.718310 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:41:31.718313 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:41:31.718330 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:41:31.718426 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:31.718432 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:31.719526 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:31.719529 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:31.721211 (+ 1682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:31.721218 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:31.722642 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:31.722645 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:31.722729 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:31.722748 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 04:41:31.722808 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 04:41:31.722903 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":237,"thread_start_us":92,"threads_started":1}
I20260925 04:41:31.723476 14638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:31.717437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45092 (local address 127.0.98.1:34513)
0925 04:41:31.717775 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0925 04:41:31.717780 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:41:31.717797 (+ 17us) server_negotiation.cc:407] Connection header received
0925 04:41:31.717863 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:41:31.717866 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:41:31.718047 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:41:31.718184 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:41:31.718547 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:31.719410 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:31.721604 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:31.722364 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:31.722461 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:31.722587 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 04:41:31.723272 (+ 685us) server_negotiation.cc:299] Negotiation successful
0925 04:41:31.723371 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":67,"threads_started":1}
I20260925 04:41:31.726280 14288 tablet_service.cc:1467] Processing CreateTablet for tablet 01a3250bb5fc4bca9dd6c2d8e45e1506 (DEFAULT_TABLE table=test-table [id=57c74e7c37a84290baf9245e7bf31608]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:41:31.726285 14556 tablet_service.cc:1467] Processing CreateTablet for tablet 01a3250bb5fc4bca9dd6c2d8e45e1506 (DEFAULT_TABLE table=test-table [id=57c74e7c37a84290baf9245e7bf31608]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:41:31.726285 14422 tablet_service.cc:1467] Processing CreateTablet for tablet 01a3250bb5fc4bca9dd6c2d8e45e1506 (DEFAULT_TABLE table=test-table [id=57c74e7c37a84290baf9245e7bf31608]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:41:31.727264 14288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01a3250bb5fc4bca9dd6c2d8e45e1506. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:41:31.727293 14556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01a3250bb5fc4bca9dd6c2d8e45e1506. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:41:31.727288 14422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01a3250bb5fc4bca9dd6c2d8e45e1506. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:41:31.734452 14639 tablet_bootstrap.cc:492] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P 7a781f32e0144cbf99f8bbff69c3fdac: Bootstrap starting.
I20260925 04:41:31.734912 14640 tablet_bootstrap.cc:492] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P 08cb310fd578413c950d87af40935374: Bootstrap starting.
I20260925 04:41:31.735920 14641 tablet_bootstrap.cc:492] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P fceaa38dcc7f4a8fae17e2252f431b15: Bootstrap starting.
I20260925 04:41:31.736997 14639 tablet_bootstrap.cc:654] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P 7a781f32e0144cbf99f8bbff69c3fdac: Neither blocks nor log segments found. Creating new log.
I20260925 04:41:31.737375 14640 tablet_bootstrap.cc:654] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P 08cb310fd578413c950d87af40935374: Neither blocks nor log segments found. Creating new log.
I20260925 04:41:31.737666 14641 tablet_bootstrap.cc:654] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P fceaa38dcc7f4a8fae17e2252f431b15: Neither blocks nor log segments found. Creating new log.
I20260925 04:41:31.737941 14639 log.cc:824] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P 7a781f32e0144cbf99f8bbff69c3fdac: Log is configured to *not* fsync() on all Append() calls
I20260925 04:41:31.738183 14640 log.cc:824] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P 08cb310fd578413c950d87af40935374: Log is configured to *not* fsync() on all Append() calls
I20260925 04:41:31.738772 14641 log.cc:824] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P fceaa38dcc7f4a8fae17e2252f431b15: Log is configured to *not* fsync() on all Append() calls
I20260925 04:41:31.740291 14639 tablet_bootstrap.cc:492] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P 7a781f32e0144cbf99f8bbff69c3fdac: No bootstrap required, opened a new log
I20260925 04:41:31.740291 14640 tablet_bootstrap.cc:492] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P 08cb310fd578413c950d87af40935374: No bootstrap required, opened a new log
I20260925 04:41:31.740507 14640 ts_tablet_manager.cc:1397] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P 08cb310fd578413c950d87af40935374: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260925 04:41:31.740515 14639 ts_tablet_manager.cc:1397] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P 7a781f32e0144cbf99f8bbff69c3fdac: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.004s
I20260925 04:41:31.740933 14641 tablet_bootstrap.cc:492] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P fceaa38dcc7f4a8fae17e2252f431b15: No bootstrap required, opened a new log
I20260925 04:41:31.741101 14641 ts_tablet_manager.cc:1397] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P fceaa38dcc7f4a8fae17e2252f431b15: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 04:41:31.743427 14640 raft_consensus.cc:348] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P 08cb310fd578413c950d87af40935374 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fceaa38dcc7f4a8fae17e2252f431b15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44101 } } peers { permanent_uuid: "08cb310fd578413c950d87af40935374" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41513 } } peers { permanent_uuid: "7a781f32e0144cbf99f8bbff69c3fdac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34513 } }
I20260925 04:41:31.743700 14640 raft_consensus.cc:374] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P 08cb310fd578413c950d87af40935374 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:41:31.743754 14640 raft_consensus.cc:729] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P 08cb310fd578413c950d87af40935374 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08cb310fd578413c950d87af40935374, State: Initialized, Role: FOLLOWER
I20260925 04:41:31.744185 14640 consensus_queue.cc:260] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P 08cb310fd578413c950d87af40935374 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fceaa38dcc7f4a8fae17e2252f431b15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44101 } } peers { permanent_uuid: "08cb310fd578413c950d87af40935374" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41513 } } peers { permanent_uuid: "7a781f32e0144cbf99f8bbff69c3fdac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34513 } }
I20260925 04:41:31.744890 14641 raft_consensus.cc:348] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P fceaa38dcc7f4a8fae17e2252f431b15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fceaa38dcc7f4a8fae17e2252f431b15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44101 } } peers { permanent_uuid: "08cb310fd578413c950d87af40935374" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41513 } } peers { permanent_uuid: "7a781f32e0144cbf99f8bbff69c3fdac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34513 } }
I20260925 04:41:31.745149 14640 ts_tablet_manager.cc:1428] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P 08cb310fd578413c950d87af40935374: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 04:41:31.745172 14641 raft_consensus.cc:374] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P fceaa38dcc7f4a8fae17e2252f431b15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:41:31.745220 14641 raft_consensus.cc:729] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P fceaa38dcc7f4a8fae17e2252f431b15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fceaa38dcc7f4a8fae17e2252f431b15, State: Initialized, Role: FOLLOWER
I20260925 04:41:31.745376 14639 raft_consensus.cc:348] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P 7a781f32e0144cbf99f8bbff69c3fdac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fceaa38dcc7f4a8fae17e2252f431b15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44101 } } peers { permanent_uuid: "08cb310fd578413c950d87af40935374" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41513 } } peers { permanent_uuid: "7a781f32e0144cbf99f8bbff69c3fdac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34513 } }
I20260925 04:41:31.745560 14488 heartbeater.cc:503] Master 127.0.98.62:46403 was elected leader, sending a full tablet report...
I20260925 04:41:31.745666 14639 raft_consensus.cc:374] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P 7a781f32e0144cbf99f8bbff69c3fdac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:41:31.745738 14639 raft_consensus.cc:729] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P 7a781f32e0144cbf99f8bbff69c3fdac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a781f32e0144cbf99f8bbff69c3fdac, State: Initialized, Role: FOLLOWER
I20260925 04:41:31.745676 14641 consensus_queue.cc:260] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P fceaa38dcc7f4a8fae17e2252f431b15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fceaa38dcc7f4a8fae17e2252f431b15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44101 } } peers { permanent_uuid: "08cb310fd578413c950d87af40935374" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41513 } } peers { permanent_uuid: "7a781f32e0144cbf99f8bbff69c3fdac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34513 } }
I20260925 04:41:31.746181 14639 consensus_queue.cc:260] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P 7a781f32e0144cbf99f8bbff69c3fdac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fceaa38dcc7f4a8fae17e2252f431b15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44101 } } peers { permanent_uuid: "08cb310fd578413c950d87af40935374" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41513 } } peers { permanent_uuid: "7a781f32e0144cbf99f8bbff69c3fdac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34513 } }
I20260925 04:41:31.746358 14622 heartbeater.cc:503] Master 127.0.98.62:46403 was elected leader, sending a full tablet report...
I20260925 04:41:31.746660 14641 ts_tablet_manager.cc:1428] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P fceaa38dcc7f4a8fae17e2252f431b15: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 04:41:31.747162 14639 ts_tablet_manager.cc:1428] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P 7a781f32e0144cbf99f8bbff69c3fdac: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260925 04:41:31.747346 14354 heartbeater.cc:503] Master 127.0.98.62:46403 was elected leader, sending a full tablet report...
W20260925 04:41:31.914005 14623 tablet.cc:2369] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P fceaa38dcc7f4a8fae17e2252f431b15: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 04:41:31.935935 14489 tablet.cc:2369] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P 08cb310fd578413c950d87af40935374: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 04:41:31.938444 14355 tablet.cc:2369] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P 7a781f32e0144cbf99f8bbff69c3fdac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:41:32.030187 14645 raft_consensus.cc:482] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P 08cb310fd578413c950d87af40935374 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:41:32.030391 14645 raft_consensus.cc:504] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P 08cb310fd578413c950d87af40935374 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fceaa38dcc7f4a8fae17e2252f431b15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44101 } } peers { permanent_uuid: "08cb310fd578413c950d87af40935374" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41513 } } peers { permanent_uuid: "7a781f32e0144cbf99f8bbff69c3fdac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34513 } }
I20260925 04:41:32.031558 14645 leader_election.cc:288] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P 08cb310fd578413c950d87af40935374 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fceaa38dcc7f4a8fae17e2252f431b15 (127.0.98.3:44101), 7a781f32e0144cbf99f8bbff69c3fdac (127.0.98.1:34513)
I20260925 04:41:32.034399 14646 raft_consensus.cc:482] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P fceaa38dcc7f4a8fae17e2252f431b15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:41:32.034605 14646 raft_consensus.cc:504] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P fceaa38dcc7f4a8fae17e2252f431b15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fceaa38dcc7f4a8fae17e2252f431b15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44101 } } peers { permanent_uuid: "08cb310fd578413c950d87af40935374" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41513 } } peers { permanent_uuid: "7a781f32e0144cbf99f8bbff69c3fdac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34513 } }
I20260925 04:41:32.036911 14635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:32.031900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52437 (local address 127.0.98.3:44101)
0925 04:41:32.032024 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 04:41:32.032029 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:41:32.032264 (+ 235us) server_negotiation.cc:407] Connection header received
0925 04:41:32.032406 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:41:32.032409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:41:32.032462 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:41:32.032547 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:41:32.033015 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:32.033785 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:32.035570 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:32.036203 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:32.036300 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:32.036363 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 04:41:32.036672 (+ 309us) server_negotiation.cc:299] Negotiation successful
0925 04:41:32.036752 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 04:41:32.036919 14649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:32.031813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44101 (local address 127.0.98.2:52437)
0925 04:41:32.032147 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:32.032162 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:41:32.032297 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:41:32.032599 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:41:32.032607 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:41:32.032648 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:41:32.032829 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:32.032835 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:32.033923 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:32.033928 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:32.035430 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:32.035444 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:32.036432 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:32.036436 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:32.036545 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:32.036572 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 04:41:32.036652 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 04:41:32.036743 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":179,"spinlock_wait_cycles":32384,"thread_start_us":95,"threads_started":1}
I20260925 04:41:32.037050 14638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:32.031990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44159 (local address 127.0.98.1:34513)
0925 04:41:32.032114 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 04:41:32.032118 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:41:32.032316 (+ 198us) server_negotiation.cc:407] Connection header received
0925 04:41:32.032406 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:41:32.032409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:41:32.032462 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:41:32.032546 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:41:32.032987 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:32.033796 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:32.035726 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:32.036393 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:32.036499 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:32.036558 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 04:41:32.036860 (+ 302us) server_negotiation.cc:299] Negotiation successful
0925 04:41:32.036937 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 04:41:32.036789 14646 leader_election.cc:288] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P fceaa38dcc7f4a8fae17e2252f431b15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08cb310fd578413c950d87af40935374 (127.0.98.2:41513), 7a781f32e0144cbf99f8bbff69c3fdac (127.0.98.1:34513)
I20260925 04:41:32.037624 14648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:32.031884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34513 (local address 127.0.98.2:44159)
0925 04:41:32.032238 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:32.032249 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 04:41:32.032319 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:41:32.032599 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:41:32.032603 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:41:32.032649 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:41:32.032823 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:32.032832 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:32.033923 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:32.033927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:32.035430 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:32.035444 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:32.036614 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:32.036618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:32.036721 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:32.036743 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 04:41:32.037371 (+ 628us) client_negotiation.cc:240] Negotiation successful
0925 04:41:32.037471 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":60,"spinlock_wait_cycles":9216,"thread_start_us":90,"threads_started":1}
I20260925 04:41:32.037621 14576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01a3250bb5fc4bca9dd6c2d8e45e1506" candidate_uuid: "08cb310fd578413c950d87af40935374" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fceaa38dcc7f4a8fae17e2252f431b15" is_pre_election: true
I20260925 04:41:32.037868 14576 raft_consensus.cc:2454] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P fceaa38dcc7f4a8fae17e2252f431b15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08cb310fd578413c950d87af40935374 in term 0.
I20260925 04:41:32.038224 14308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01a3250bb5fc4bca9dd6c2d8e45e1506" candidate_uuid: "08cb310fd578413c950d87af40935374" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a781f32e0144cbf99f8bbff69c3fdac" is_pre_election: true
I20260925 04:41:32.038338 14378 leader_election.cc:302] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P 08cb310fd578413c950d87af40935374 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08cb310fd578413c950d87af40935374, fceaa38dcc7f4a8fae17e2252f431b15; no voters:
I20260925 04:41:32.038525 14308 raft_consensus.cc:2454] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P 7a781f32e0144cbf99f8bbff69c3fdac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08cb310fd578413c950d87af40935374 in term 0.
I20260925 04:41:32.038645 14645 raft_consensus.cc:2781] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P 08cb310fd578413c950d87af40935374 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:41:32.038712 14645 raft_consensus.cc:482] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P 08cb310fd578413c950d87af40935374 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:41:32.038746 14645 raft_consensus.cc:3037] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P 08cb310fd578413c950d87af40935374 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:41:32.040143 14645 raft_consensus.cc:504] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P 08cb310fd578413c950d87af40935374 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fceaa38dcc7f4a8fae17e2252f431b15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44101 } } peers { permanent_uuid: "08cb310fd578413c950d87af40935374" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41513 } } peers { permanent_uuid: "7a781f32e0144cbf99f8bbff69c3fdac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34513 } }
I20260925 04:41:32.040557 14645 leader_election.cc:288] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P 08cb310fd578413c950d87af40935374 [CANDIDATE]: Term 1 election: Requested vote from peers fceaa38dcc7f4a8fae17e2252f431b15 (127.0.98.3:44101), 7a781f32e0144cbf99f8bbff69c3fdac (127.0.98.1:34513)
I20260925 04:41:32.040906 14308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01a3250bb5fc4bca9dd6c2d8e45e1506" candidate_uuid: "08cb310fd578413c950d87af40935374" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a781f32e0144cbf99f8bbff69c3fdac"
I20260925 04:41:32.040937 14576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01a3250bb5fc4bca9dd6c2d8e45e1506" candidate_uuid: "08cb310fd578413c950d87af40935374" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fceaa38dcc7f4a8fae17e2252f431b15"
I20260925 04:41:32.041038 14576 raft_consensus.cc:3037] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P fceaa38dcc7f4a8fae17e2252f431b15 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:41:32.041038 14308 raft_consensus.cc:3037] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P 7a781f32e0144cbf99f8bbff69c3fdac [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:41:32.041503 14624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:32.036045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34513 (local address 127.0.98.3:46447)
0925 04:41:32.036323 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:32.036338 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:41:32.036412 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:41:32.038079 (+ 1667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:41:32.038082 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:41:32.038102 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:41:32.038202 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:32.038208 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:32.039243 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:32.039246 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:32.040149 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:32.040159 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:32.041125 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:32.041129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:32.041227 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:32.041249 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 04:41:32.041291 (+ 42us) client_negotiation.cc:240] Negotiation successful
0925 04:41:32.041372 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":1}
I20260925 04:41:32.041925 14651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:32.036564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46447 (local address 127.0.98.1:34513)
0925 04:41:32.037142 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0925 04:41:32.037166 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0925 04:41:32.037180 (+ 14us) server_negotiation.cc:407] Connection header received
0925 04:41:32.037222 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:41:32.037225 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:41:32.037272 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:41:32.038304 (+ 1032us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:41:32.038405 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:32.039112 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:32.040322 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:32.040986 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:32.041672 (+ 686us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:32.041734 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 04:41:32.041767 (+ 33us) server_negotiation.cc:299] Negotiation successful
0925 04:41:32.041822 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":124,"threads_started":1}
I20260925 04:41:32.042289 14576 raft_consensus.cc:2454] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P fceaa38dcc7f4a8fae17e2252f431b15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08cb310fd578413c950d87af40935374 in term 1.
I20260925 04:41:32.042287 14307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01a3250bb5fc4bca9dd6c2d8e45e1506" candidate_uuid: "fceaa38dcc7f4a8fae17e2252f431b15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a781f32e0144cbf99f8bbff69c3fdac" is_pre_election: true
I20260925 04:41:32.042550 14650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:32.036077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41513 (local address 127.0.98.3:44321)
0925 04:41:32.037389 (+ 1312us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:32.037403 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:41:32.037505 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:41:32.037799 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:41:32.037802 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:41:32.037818 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:41:32.038035 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:32.038043 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:32.039061 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:32.039066 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:32.040239 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:32.040250 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:32.042098 (+ 1848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:32.042102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:32.042197 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:32.042223 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 04:41:32.042331 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 04:41:32.042406 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1238,"thread_start_us":112,"threads_started":1}
I20260925 04:41:32.042553 14636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:32.036487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44321 (local address 127.0.98.2:41513)
0925 04:41:32.036596 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 04:41:32.036600 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:41:32.037485 (+ 885us) server_negotiation.cc:407] Connection header received
0925 04:41:32.037581 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:41:32.037584 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:41:32.037629 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:41:32.037701 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:41:32.038158 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:32.038820 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:32.040356 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:32.041000 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:32.041090 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:32.041151 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 04:41:32.042360 (+ 1209us) server_negotiation.cc:299] Negotiation successful
0925 04:41:32.042447 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 04:41:32.042595 14378 leader_election.cc:302] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P 08cb310fd578413c950d87af40935374 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08cb310fd578413c950d87af40935374, fceaa38dcc7f4a8fae17e2252f431b15; no voters:
I20260925 04:41:32.042295 14308 raft_consensus.cc:2454] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P 7a781f32e0144cbf99f8bbff69c3fdac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08cb310fd578413c950d87af40935374 in term 1.
I20260925 04:41:32.043136 14442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01a3250bb5fc4bca9dd6c2d8e45e1506" candidate_uuid: "fceaa38dcc7f4a8fae17e2252f431b15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08cb310fd578413c950d87af40935374" is_pre_election: true
I20260925 04:41:32.043411 14442 raft_consensus.cc:2379] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P 08cb310fd578413c950d87af40935374 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fceaa38dcc7f4a8fae17e2252f431b15 in current term 1: Already voted for candidate 08cb310fd578413c950d87af40935374 in this term.
I20260925 04:41:32.043651 14645 raft_consensus.cc:2781] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P 08cb310fd578413c950d87af40935374 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:41:32.043882 14645 raft_consensus.cc:686] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P 08cb310fd578413c950d87af40935374 [term 1 LEADER]: Becoming Leader. State: Replica: 08cb310fd578413c950d87af40935374, State: Running, Role: LEADER
I20260925 04:41:32.043949 14509 leader_election.cc:302] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P fceaa38dcc7f4a8fae17e2252f431b15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fceaa38dcc7f4a8fae17e2252f431b15; no voters: 08cb310fd578413c950d87af40935374, 7a781f32e0144cbf99f8bbff69c3fdac
I20260925 04:41:32.044240 14646 raft_consensus.cc:2726] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P fceaa38dcc7f4a8fae17e2252f431b15 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 04:41:32.044245 14645 consensus_queue.cc:237] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P 08cb310fd578413c950d87af40935374 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fceaa38dcc7f4a8fae17e2252f431b15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44101 } } peers { permanent_uuid: "08cb310fd578413c950d87af40935374" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41513 } } peers { permanent_uuid: "7a781f32e0144cbf99f8bbff69c3fdac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34513 } }
I20260925 04:41:32.047533 14160 catalog_manager.cc:5495] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P 08cb310fd578413c950d87af40935374 reported cstate change: term changed from 0 to 1, leader changed from <none> to 08cb310fd578413c950d87af40935374 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "08cb310fd578413c950d87af40935374" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fceaa38dcc7f4a8fae17e2252f431b15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08cb310fd578413c950d87af40935374" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a781f32e0144cbf99f8bbff69c3fdac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34513 } health_report { overall_health: UNKNOWN } } }
I20260925 04:41:32.089277 14636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:32.085562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50504 (local address 127.0.98.2:41513)
0925 04:41:32.085716 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 04:41:32.085721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:41:32.085800 (+ 79us) server_negotiation.cc:407] Connection header received
0925 04:41:32.085883 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:41:32.085885 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:41:32.085939 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:41:32.086048 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:41:32.086317 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:32.086853 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:32.087769 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:32.087989 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:32.088082 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:32.088410 (+ 328us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:41:32.088876 (+ 466us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:41:32.088984 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 04:41:32.089064 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 04:41:32.089123 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 04:41:32.095763 14308 raft_consensus.cc:1261] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P 7a781f32e0144cbf99f8bbff69c3fdac [term 1 FOLLOWER]: Refusing update from remote peer 08cb310fd578413c950d87af40935374: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:41:32.095870 14576 raft_consensus.cc:1261] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P fceaa38dcc7f4a8fae17e2252f431b15 [term 1 FOLLOWER]: Refusing update from remote peer 08cb310fd578413c950d87af40935374: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:41:32.096508 14645 consensus_queue.cc:1035] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P 08cb310fd578413c950d87af40935374 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fceaa38dcc7f4a8fae17e2252f431b15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:41:32.096920 14652 consensus_queue.cc:1035] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P 08cb310fd578413c950d87af40935374 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a781f32e0144cbf99f8bbff69c3fdac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:41:32.115805 14160 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:41:32.116022 14160 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:41:32.118623 14160 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5ad9c8d894c349b58d60ea2e39f3d704: Sending DeleteTablet for 3 replicas of tablet 01a3250bb5fc4bca9dd6c2d8e45e1506
I20260925 04:41:32.119441 14422 tablet_service.cc:1514] Processing DeleteTablet for tablet 01a3250bb5fc4bca9dd6c2d8e45e1506 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:41:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50490
I20260925 04:41:32.119447 14556 tablet_service.cc:1514] Processing DeleteTablet for tablet 01a3250bb5fc4bca9dd6c2d8e45e1506 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:41:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40446
I20260925 04:41:32.119952 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 14225
I20260925 04:41:32.121132 14667 tablet_replica.cc:330] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P 08cb310fd578413c950d87af40935374: stopping tablet replica
I20260925 04:41:32.121483 14667 raft_consensus.cc:2229] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P 08cb310fd578413c950d87af40935374 [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:41:32.121599 14666 tablet_replica.cc:330] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P fceaa38dcc7f4a8fae17e2252f431b15: stopping tablet replica
I20260925 04:41:32.121901 14666 raft_consensus.cc:2229] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P fceaa38dcc7f4a8fae17e2252f431b15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:41:32.121937 14667 raft_consensus.cc:2258] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P 08cb310fd578413c950d87af40935374 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:41:32.122215 14666 raft_consensus.cc:2258] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P fceaa38dcc7f4a8fae17e2252f431b15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:41:32.123926 14667 ts_tablet_manager.cc:1905] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P 08cb310fd578413c950d87af40935374: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:41:32.124379 14666 ts_tablet_manager.cc:1905] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P fceaa38dcc7f4a8fae17e2252f431b15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:41:32.126845 14667 ts_tablet_manager.cc:1918] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P 08cb310fd578413c950d87af40935374: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:41:32.126993 14667 log.cc:1196] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P 08cb310fd578413c950d87af40935374: Deleting WAL directory at /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-1/wal/wals/01a3250bb5fc4bca9dd6c2d8e45e1506
I20260925 04:41:32.127099 14666 ts_tablet_manager.cc:1918] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P fceaa38dcc7f4a8fae17e2252f431b15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:41:32.127272 14666 log.cc:1196] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P fceaa38dcc7f4a8fae17e2252f431b15: Deleting WAL directory at /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-2/wal/wals/01a3250bb5fc4bca9dd6c2d8e45e1506
I20260925 04:41:32.127327 14667 ts_tablet_manager.cc:1939] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P 08cb310fd578413c950d87af40935374: Deleting consensus metadata
W20260925 04:41:32.127324 14148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34513 (error 108)
I20260925 04:41:32.127996 14666 ts_tablet_manager.cc:1939] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P fceaa38dcc7f4a8fae17e2252f431b15: Deleting consensus metadata
I20260925 04:41:32.128227 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 14359
I20260925 04:41:32.130337 14150 catalog_manager.cc:4841] TS fceaa38dcc7f4a8fae17e2252f431b15 (127.0.98.3:44101): tablet 01a3250bb5fc4bca9dd6c2d8e45e1506 (table test-table [id=57c74e7c37a84290baf9245e7bf31608]) successfully deleted
I20260925 04:41:32.130616 14634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:32.130338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34513 (local address 127.0.0.1:45098)
0925 04:41:32.130466 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:32.130538 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 04:41:32.130771 14148 catalog_manager.cc:4584] TS 7a781f32e0144cbf99f8bbff69c3fdac (127.0.98.1:34513): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 01a3250bb5fc4bca9dd6c2d8e45e1506: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34513: connect: Connection refused (error 111)
I20260925 04:41:32.133863 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 14493
I20260925 04:41:32.134111 14634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:32.133847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41513 (local address 127.0.0.1:50514)
0925 04:41:32.133966 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:32.134018 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:41513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 04:41:32.138921 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 14131
I20260925 04:41:32.145044 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46403
--webserver_interface=127.0.98.62
--webserver_port=34573
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46403
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:41:32.260149 14669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:41:32.260452 14669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:41:32.260493 14669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:41:32.264648 14669 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:41:32.264727 14669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:41:32.264746 14669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:41:32.264762 14669 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:41:32.264777 14669 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:41:32.270349 14669 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46403
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=34573
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:41:32.271502 14669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:41:32.272569 14669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:41:32.285391 14676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:41:32.285507 14669 server_base.cc:1034] running on GCE node
W20260925 04:41:32.285473 14678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:41:32.285390 14675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:41:32.286388 14669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:41:32.287631 14669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:41:32.288820 14669 hybrid_clock.cc:648] HybridClock initialized: now 1790311292288785 us; error 49 us; skew 500 ppm
Sep 25 04:41:32 dist-test-slave-dc95 krb5kdc[14118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311292, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:41:32.291785 14669 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 04:41:32.292824 14669 webserver.cc:460] Webserver started at http://127.0.98.62:34573/ using document root <none> and password file <none>
I20260925 04:41:32.293411 14669 fs_manager.cc:359] Metadata directory not provided
I20260925 04:41:32.293495 14669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:41:32.297703 14669 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:41:32.300122 14685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:41:32.301704 14669 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:41:32.301855 14669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/master-0/wal
uuid: "5ad9c8d894c349b58d60ea2e39f3d704"
format_stamp: "Formatted at 2026-09-25 04:41:30 on dist-test-slave-dc95"
server_key: "1a99b4b373f3cbe241fa387b7cee2418"
server_key_iv: "a44e137621847188cdc1c3967ae690c3"
server_key_version: "encryptionkey@0"
I20260925 04:41:32.302500 14669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:41:32.321656 14669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:41:32.325227 14669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:41:32.325428 14669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:41:32.332631 14669 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46403
I20260925 04:41:32.332899 14737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46403 every 8 connection(s)
I20260925 04:41:32.333770 14669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
I20260925 04:41:32.338533 14738 sys_catalog.cc:263] Verifying existing consensus state
I20260925 04:41:32.340515 14738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ad9c8d894c349b58d60ea2e39f3d704: Bootstrap starting.
I20260925 04:41:32.342137 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 14669
I20260925 04:41:32.342594 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:34513
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:46403
--webserver_port=35049
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:41:32.362792 14738 log.cc:824] T 00000000000000000000000000000000 P 5ad9c8d894c349b58d60ea2e39f3d704: Log is configured to *not* fsync() on all Append() calls
I20260925 04:41:32.374316 14738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ad9c8d894c349b58d60ea2e39f3d704: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 04:41:32.374804 14738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ad9c8d894c349b58d60ea2e39f3d704: Bootstrap complete.
I20260925 04:41:32.380218 14738 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ad9c8d894c349b58d60ea2e39f3d704 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad9c8d894c349b58d60ea2e39f3d704" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46403 } }
I20260925 04:41:32.380748 14738 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ad9c8d894c349b58d60ea2e39f3d704 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ad9c8d894c349b58d60ea2e39f3d704, State: Initialized, Role: FOLLOWER
I20260925 04:41:32.381305 14738 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ad9c8d894c349b58d60ea2e39f3d704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad9c8d894c349b58d60ea2e39f3d704" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46403 } }
I20260925 04:41:32.381454 14738 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ad9c8d894c349b58d60ea2e39f3d704 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:41:32.381536 14738 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ad9c8d894c349b58d60ea2e39f3d704 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:41:32.381632 14738 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ad9c8d894c349b58d60ea2e39f3d704 [term 1 FOLLOWER]: Advancing to term 2
I20260925 04:41:32.384191 14738 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ad9c8d894c349b58d60ea2e39f3d704 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad9c8d894c349b58d60ea2e39f3d704" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46403 } }
I20260925 04:41:32.384603 14738 leader_election.cc:302] T 00000000000000000000000000000000 P 5ad9c8d894c349b58d60ea2e39f3d704 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ad9c8d894c349b58d60ea2e39f3d704; no voters:
I20260925 04:41:32.385003 14738 leader_election.cc:288] T 00000000000000000000000000000000 P 5ad9c8d894c349b58d60ea2e39f3d704 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 04:41:32.385131 14742 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ad9c8d894c349b58d60ea2e39f3d704 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 04:41:32.385390 14742 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ad9c8d894c349b58d60ea2e39f3d704 [term 2 LEADER]: Becoming Leader. State: Replica: 5ad9c8d894c349b58d60ea2e39f3d704, State: Running, Role: LEADER
I20260925 04:41:32.385747 14742 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ad9c8d894c349b58d60ea2e39f3d704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad9c8d894c349b58d60ea2e39f3d704" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46403 } }
I20260925 04:41:32.386298 14738 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ad9c8d894c349b58d60ea2e39f3d704 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:41:32.387200 14744 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ad9c8d894c349b58d60ea2e39f3d704 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ad9c8d894c349b58d60ea2e39f3d704. Latest consensus state: current_term: 2 leader_uuid: "5ad9c8d894c349b58d60ea2e39f3d704" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad9c8d894c349b58d60ea2e39f3d704" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46403 } } }
I20260925 04:41:32.387210 14743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ad9c8d894c349b58d60ea2e39f3d704 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5ad9c8d894c349b58d60ea2e39f3d704" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad9c8d894c349b58d60ea2e39f3d704" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46403 } } }
I20260925 04:41:32.387341 14743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ad9c8d894c349b58d60ea2e39f3d704 [sys.catalog]: This master's current role is: LEADER
I20260925 04:41:32.387341 14744 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ad9c8d894c349b58d60ea2e39f3d704 [sys.catalog]: This master's current role is: LEADER
I20260925 04:41:32.387756 14751 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:41:32.392694 14751 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:41:32.393509 14751 catalog_manager.cc:1259] Loaded cluster ID: 1235377a389c4d639f6e15e42f920f1f
I20260925 04:41:32.393570 14751 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:41:32.394855 14751 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260925 04:41:32.397945 14751 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260925 04:41:32.402818 14751 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:41:32.403765 14751 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5ad9c8d894c349b58d60ea2e39f3d704: Loaded TSK: 0
I20260925 04:41:32.404944 14751 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 04:41:32.481884 14740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:41:32.482146 14740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:41:32.482210 14740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:41:32.486411 14740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:41:32.486506 14740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:41:32.486593 14740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 04:41:32.492321 14740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:34513
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35049
--tserver_master_addrs=127.0.98.62:46403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:41:32.493500 14740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:41:32.494578 14740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:41:32.505877 14766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:41:32.506027 14768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:41:32.505884 14765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:41:32.506312 14740 server_base.cc:1034] running on GCE node
I20260925 04:41:32.506834 14740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:41:32.507515 14740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:41:32.508701 14740 hybrid_clock.cc:648] HybridClock initialized: now 1790311292508664 us; error 55 us; skew 500 ppm
Sep 25 04:41:32 dist-test-slave-dc95 krb5kdc[14118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311292, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:41:32.511744 14740 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 04:41:32.512735 14740 webserver.cc:460] Webserver started at http://127.0.98.1:35049/ using document root <none> and password file <none>
I20260925 04:41:32.513298 14740 fs_manager.cc:359] Metadata directory not provided
I20260925 04:41:32.513370 14740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:41:32.517699 14740 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:41:32.520165 14775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:41:32.521548 14740 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:41:32.521669 14740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-0/wal
uuid: "7a781f32e0144cbf99f8bbff69c3fdac"
format_stamp: "Formatted at 2026-09-25 04:41:31 on dist-test-slave-dc95"
server_key: "6893d856fc402642507b48120ed8ef4a"
server_key_iv: "3424d3d43039b8b6596f7353b3cd293a"
server_key_version: "encryptionkey@0"
I20260925 04:41:32.522284 14740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:41:32.547848 14740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:41:32.551297 14740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:41:32.551476 14740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:41:32.552120 14740 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:41:32.553265 14782 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 04:41:32.557493 14740 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 04:41:32.557575 14740 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260925 04:41:32.557634 14740 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 04:41:32.559403 14740 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 04:41:32.559459 14740 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260925 04:41:32.559868 14782 tablet_bootstrap.cc:492] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P 7a781f32e0144cbf99f8bbff69c3fdac: Bootstrap starting.
I20260925 04:41:32.569434 14740 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34513
I20260925 04:41:32.569550 14889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34513 every 8 connection(s)
I20260925 04:41:32.570541 14740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
I20260925 04:41:32.578390 14782 log.cc:824] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P 7a781f32e0144cbf99f8bbff69c3fdac: Log is configured to *not* fsync() on all Append() calls
I20260925 04:41:32.580844 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 14740
I20260925 04:41:32.581312 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:41513
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:46403
--webserver_port=36097
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 04:41:32 dist-test-slave-dc95 krb5kdc[14118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311292, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:41:32.587395 14782 tablet_bootstrap.cc:492] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P 7a781f32e0144cbf99f8bbff69c3fdac: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 04:41:32.587889 14782 tablet_bootstrap.cc:492] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P 7a781f32e0144cbf99f8bbff69c3fdac: Bootstrap complete.
I20260925 04:41:32.589128 14782 ts_tablet_manager.cc:1397] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P 7a781f32e0144cbf99f8bbff69c3fdac: Time spent bootstrapping tablet: real 0.030s user 0.024s sys 0.004s
I20260925 04:41:32.589447 14893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:32.572691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.1:47715)
0925 04:41:32.573324 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:32.573379 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 04:41:32.574224 (+ 845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:41:32.575797 (+ 1573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:41:32.575805 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:41:32.576187 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:41:32.576704 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:32.576720 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:32.578062 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:32.578065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:32.579098 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:32.579111 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:32.580213 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:32.580216 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:32.580395 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:32.581100 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:41:32.581124 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:41:32.583067 (+ 1943us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:41:32.585622 (+ 2555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:41:32.585629 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:41:32.585646 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:41:32.586041 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:41:32.586471 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:41:32.586475 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:41:32.586477 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:41:32.586697 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:41:32.587552 (+ 855us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:41:32.587558 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:41:32.587946 (+ 388us) client_negotiation.cc:769] Sending connection context
0925 04:41:32.588128 (+ 182us) client_negotiation.cc:240] Negotiation successful
0925 04:41:32.588478 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":165,"threads_started":1}
I20260925 04:41:32.591219 14892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:32.572789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47715 (local address 127.0.98.62:46403)
0925 04:41:32.573177 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0925 04:41:32.573195 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 04:41:32.573521 (+ 326us) server_negotiation.cc:407] Connection header received
0925 04:41:32.574726 (+ 1205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:41:32.574744 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:41:32.575263 (+ 519us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:41:32.575729 (+ 466us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:41:32.576893 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:32.577877 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:32.579252 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:32.580030 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:32.580220 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:32.583313 (+ 3093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:41:32.583320 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:41:32.583346 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:41:32.583385 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:41:32.585459 (+ 2074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:32.586165 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:32.586172 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:32.586178 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:32.586357 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:32.586815 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:32.586818 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:32.586820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:32.587414 (+ 594us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:41:32.589842 (+ 2428us) server_negotiation.cc:1035] Waiting for connection context
0925 04:41:32.590018 (+ 176us) server_negotiation.cc:299] Negotiation successful
0925 04:41:32.590359 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":120,"threads_started":1}
I20260925 04:41:32.592185 14782 raft_consensus.cc:348] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P 7a781f32e0144cbf99f8bbff69c3fdac [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fceaa38dcc7f4a8fae17e2252f431b15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44101 } } peers { permanent_uuid: "08cb310fd578413c950d87af40935374" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41513 } } peers { permanent_uuid: "7a781f32e0144cbf99f8bbff69c3fdac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34513 } }
I20260925 04:41:32.592597 14782 raft_consensus.cc:729] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P 7a781f32e0144cbf99f8bbff69c3fdac [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a781f32e0144cbf99f8bbff69c3fdac, State: Initialized, Role: FOLLOWER
I20260925 04:41:32.593089 14782 consensus_queue.cc:260] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P 7a781f32e0144cbf99f8bbff69c3fdac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fceaa38dcc7f4a8fae17e2252f431b15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44101 } } peers { permanent_uuid: "08cb310fd578413c950d87af40935374" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41513 } } peers { permanent_uuid: "7a781f32e0144cbf99f8bbff69c3fdac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34513 } }
I20260925 04:41:32.594044 14890 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46403
I20260925 04:41:32.594151 14782 ts_tablet_manager.cc:1428] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P 7a781f32e0144cbf99f8bbff69c3fdac: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 04:41:32.594280 14890 heartbeater.cc:464] Registering TS with master...
I20260925 04:41:32.594831 14890 heartbeater.cc:511] Master 127.0.98.62:46403 requested a full tablet report, sending...
I20260925 04:41:32.597532 14702 ts_manager.cc:194] Registered new tserver with Master: 7a781f32e0144cbf99f8bbff69c3fdac (127.0.98.1:34513)
I20260925 04:41:32.601034 14702 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47715
I20260925 04:41:32.604117 14890 heartbeater.cc:503] Master 127.0.98.62:46403 was elected leader, sending a full tablet report...
Sep 25 04:41:32 dist-test-slave-dc95 krb5kdc[14118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311292, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 04:41:32.612077 14897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:32.600411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34513 (local address 127.0.0.1:45106)
0925 04:41:32.601037 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:32.601090 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 04:41:32.601232 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:41:32.601748 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:41:32.601754 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:41:32.602208 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:41:32.602344 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:32.602353 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:32.603762 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:32.603766 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:32.604735 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:32.604748 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:32.605477 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:32.605497 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:32.605614 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:32.606320 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:41:32.606335 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:41:32.608058 (+ 1723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:41:32.610232 (+ 2174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:41:32.610240 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:41:32.610245 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:41:32.610480 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:41:32.610783 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:41:32.610785 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:41:32.610796 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:41:32.610876 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:41:32.611410 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:41:32.611416 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:41:32.611540 (+ 124us) client_negotiation.cc:769] Sending connection context
0925 04:41:32.611720 (+ 180us) client_negotiation.cc:240] Negotiation successful
0925 04:41:32.611936 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":110,"threads_started":1}
I20260925 04:41:32.612156 14898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:32.600522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45106 (local address 127.0.98.1:34513)
0925 04:41:32.601246 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0925 04:41:32.601259 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 04:41:32.601287 (+ 28us) server_negotiation.cc:407] Connection header received
0925 04:41:32.601366 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:41:32.601371 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:41:32.601581 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:41:32.601735 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:41:32.602483 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:32.603635 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:32.604846 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:32.605356 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:32.605550 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:32.608218 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:41:32.608230 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:41:32.608243 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:41:32.608275 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:41:32.610097 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:32.610593 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:32.610599 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:32.610601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:32.610696 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:32.610958 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:32.610961 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:32.610963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:32.611269 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:41:32.611412 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 04:41:32.611809 (+ 397us) server_negotiation.cc:299] Negotiation successful
0925 04:41:32.611989 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":87,"threads_started":1}
I20260925 04:41:32.613457 14824 tablet_service.cc:1514] Processing DeleteTablet for tablet 01a3250bb5fc4bca9dd6c2d8e45e1506 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:41:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45106
I20260925 04:41:32.614530 14899 tablet_replica.cc:330] stopping tablet replica
I20260925 04:41:32.614836 14899 raft_consensus.cc:2229] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P 7a781f32e0144cbf99f8bbff69c3fdac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:41:32.615106 14899 raft_consensus.cc:2258] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P 7a781f32e0144cbf99f8bbff69c3fdac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:41:32.616281 14899 ts_tablet_manager.cc:1905] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P 7a781f32e0144cbf99f8bbff69c3fdac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:41:32.619964 14899 ts_tablet_manager.cc:1918] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P 7a781f32e0144cbf99f8bbff69c3fdac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:41:32.620074 14899 log.cc:1196] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P 7a781f32e0144cbf99f8bbff69c3fdac: Deleting WAL directory at /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-0/wal/wals/01a3250bb5fc4bca9dd6c2d8e45e1506
I20260925 04:41:32.620290 14899 ts_tablet_manager.cc:1939] T 01a3250bb5fc4bca9dd6c2d8e45e1506 P 7a781f32e0144cbf99f8bbff69c3fdac: Deleting consensus metadata
I20260925 04:41:32.622090 14687 catalog_manager.cc:4841] TS 7a781f32e0144cbf99f8bbff69c3fdac (127.0.98.1:34513): tablet 01a3250bb5fc4bca9dd6c2d8e45e1506 (table test-table [id=57c74e7c37a84290baf9245e7bf31608]) successfully deleted
W20260925 04:41:32.706905 14895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:41:32.707230 14895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:41:32.707291 14895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:41:32.711714 14895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:41:32.711796 14895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:41:32.711859 14895 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 04:41:32.717571 14895 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:41513
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36097
--tserver_master_addrs=127.0.98.62:46403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14895
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:41:32.718708 14895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:41:32.719791 14895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:41:32.730296 14904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:41:32.730322 14905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:41:32.730397 14895 server_base.cc:1034] running on GCE node
W20260925 04:41:32.730322 14907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:41:32.731144 14895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:41:32.732043 14895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:41:32.733242 14895 hybrid_clock.cc:648] HybridClock initialized: now 1790311292733217 us; error 41 us; skew 500 ppm
Sep 25 04:41:32 dist-test-slave-dc95 krb5kdc[14118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311292, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:41:32.736315 14895 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 04:41:32.737320 14895 webserver.cc:460] Webserver started at http://127.0.98.2:36097/ using document root <none> and password file <none>
I20260925 04:41:32.737872 14895 fs_manager.cc:359] Metadata directory not provided
I20260925 04:41:32.737957 14895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:41:32.742200 14895 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260925 04:41:32.744825 14914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:41:32.746317 14895 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:41:32.746459 14895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-1/wal
uuid: "08cb310fd578413c950d87af40935374"
format_stamp: "Formatted at 2026-09-25 04:41:31 on dist-test-slave-dc95"
server_key: "10bb2b5d75ba0c0ee3f56bdfe9fa35c0"
server_key_iv: "51cc0695b753f9e04434fb1294aa76f6"
server_key_version: "encryptionkey@0"
I20260925 04:41:32.747061 14895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:41:32.769913 14895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:41:32.773360 14895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:41:32.773587 14895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:41:32.774217 14895 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:41:32.775111 14895 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:41:32.775162 14895 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:41:32.775218 14895 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:41:32.775247 14895 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:41:32.785763 14895 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41513
I20260925 04:41:32.785843 15027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41513 every 8 connection(s)
I20260925 04:41:32.786850 14895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
I20260925 04:41:32.789101 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 14895
I20260925 04:41:32.789510 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:44101
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:46403
--webserver_port=46641
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 04:41:32 dist-test-slave-dc95 krb5kdc[14118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311292, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:41:32.803457 14892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:32.788914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33107 (local address 127.0.98.62:46403)
0925 04:41:32.789111 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 04:41:32.789116 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:41:32.791098 (+ 1982us) server_negotiation.cc:407] Connection header received
0925 04:41:32.792508 (+ 1410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:41:32.792511 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:41:32.792603 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:41:32.792679 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:41:32.793990 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:32.794533 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:32.795768 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:32.796138 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:32.796241 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:32.799314 (+ 3073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:41:32.799318 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:41:32.799320 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:41:32.799352 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:41:32.801058 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:32.801770 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:32.801775 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:32.801776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:32.801867 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:32.802221 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:32.802224 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:32.802226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:32.802439 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:41:32.802535 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 04:41:32.803093 (+ 558us) server_negotiation.cc:299] Negotiation successful
0925 04:41:32.803282 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260925 04:41:32.804337 15030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:32.789398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.2:33107)
0925 04:41:32.790924 (+ 1526us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:32.790971 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 04:41:32.792278 (+ 1307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:41:32.792875 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:41:32.792882 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:41:32.793298 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:41:32.793821 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:32.793838 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:32.794664 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:32.794667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:32.795581 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:32.795592 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:32.796303 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:32.796307 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:32.796496 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:32.797201 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:41:32.797219 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:41:32.799098 (+ 1879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:41:32.801299 (+ 2201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:41:32.801307 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:41:32.801326 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:41:32.801660 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:41:32.801961 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:41:32.801965 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:41:32.801967 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:41:32.802124 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:41:32.802555 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:41:32.802562 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:41:32.802843 (+ 281us) client_negotiation.cc:769] Sending connection context
0925 04:41:32.803021 (+ 178us) client_negotiation.cc:240] Negotiation successful
0925 04:41:32.803393 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1299,"thread_start_us":134,"threads_started":1}
I20260925 04:41:32.805627 15028 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46403
I20260925 04:41:32.805928 15028 heartbeater.cc:464] Registering TS with master...
I20260925 04:41:32.806578 15028 heartbeater.cc:511] Master 127.0.98.62:46403 requested a full tablet report, sending...
I20260925 04:41:32.808321 14702 ts_manager.cc:194] Registered new tserver with Master: 08cb310fd578413c950d87af40935374 (127.0.98.2:41513)
I20260925 04:41:32.809127 14702 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33107
W20260925 04:41:32.911458 15031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:41:32.911790 15031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:41:32.911836 15031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:41:32.916103 15031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:41:32.916174 15031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:41:32.916235 15031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 04:41:32.923313 15031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:44101
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=46641
--tserver_master_addrs=127.0.98.62:46403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:41:32.924825 15031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:41:32.926046 15031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:41:32.936132 15039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:41:32.936111 15036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:41:32.936174 15031 server_base.cc:1034] running on GCE node
W20260925 04:41:32.936116 15037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:41:32.936843 15031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:41:32.937500 15031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:41:32.938777 15031 hybrid_clock.cc:648] HybridClock initialized: now 1790311292938710 us; error 36 us; skew 500 ppm
Sep 25 04:41:32 dist-test-slave-dc95 krb5kdc[14118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311292, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:41:32.941926 15031 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 04:41:32.943023 15031 webserver.cc:460] Webserver started at http://127.0.98.3:46641/ using document root <none> and password file <none>
I20260925 04:41:32.943841 15031 fs_manager.cc:359] Metadata directory not provided
I20260925 04:41:32.944031 15031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:41:32.948423 15031 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260925 04:41:32.951380 15046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:41:32.952940 15031 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:41:32.953073 15031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-2/wal
uuid: "fceaa38dcc7f4a8fae17e2252f431b15"
format_stamp: "Formatted at 2026-09-25 04:41:31 on dist-test-slave-dc95"
server_key: "7d91947786c8621a47da59630897f7c3"
server_key_iv: "d0ec3fc0241e0ade903599fcb32e2cdc"
server_key_version: "encryptionkey@0"
I20260925 04:41:32.953683 15031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:41:32.966503 15031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:41:32.970368 15031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:41:32.970608 15031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:41:32.971300 15031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:41:32.972242 15031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:41:32.972291 15031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:41:32.972333 15031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:41:32.972360 15031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:41:32.981331 15031 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44101
I20260925 04:41:32.981388 15159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44101 every 8 connection(s)
I20260925 04:41:32.982357 15031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
I20260925 04:41:32.986971 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 15031
Sep 25 04:41:32 dist-test-slave-dc95 krb5kdc[14118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311292, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:41:32.997365 15164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:32.987705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57334 (local address 127.0.98.62:46403)
0925 04:41:32.987960 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0925 04:41:32.987964 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:41:32.987980 (+ 16us) server_negotiation.cc:407] Connection header received
0925 04:41:32.988117 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:41:32.988120 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:41:32.988165 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:41:32.988257 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:41:32.988989 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:32.989611 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:32.990884 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:32.991339 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:32.991463 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:32.993371 (+ 1908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:41:32.993375 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:41:32.993378 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:41:32.993427 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:41:32.995251 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:32.995795 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:32.995799 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:32.995801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:32.995884 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:32.996171 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:32.996174 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:32.996176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:32.996471 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:41:32.996699 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0925 04:41:32.997032 (+ 333us) server_negotiation.cc:299] Negotiation successful
0925 04:41:32.997177 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":190,"spinlock_wait_cycles":12544,"thread_start_us":81,"threads_started":1}
I20260925 04:41:32.998883 14892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:32.984302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36697 (local address 127.0.98.62:46403)
0925 04:41:32.984454 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 04:41:32.984457 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:41:32.985439 (+ 982us) server_negotiation.cc:407] Connection header received
0925 04:41:32.986869 (+ 1430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:41:32.986873 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:41:32.986938 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:41:32.987016 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:41:32.988630 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:32.989278 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:32.990967 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:32.991339 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:32.991468 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:32.994961 (+ 3493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:41:32.994966 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:41:32.994969 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:41:32.995002 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:41:32.996551 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:32.997259 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:32.997262 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:32.997264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:32.997345 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:32.997732 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:32.997734 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:32.997736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:32.997938 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:41:32.998029 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 04:41:32.998665 (+ 636us) server_negotiation.cc:299] Negotiation successful
0925 04:41:32.998766 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":44}
I20260925 04:41:33.000038 15162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:32.984729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.3:36697)
0925 04:41:32.985261 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:32.985309 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 04:41:32.986599 (+ 1290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:41:32.987230 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:41:32.987237 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:41:32.987775 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:41:32.988487 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:32.988505 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:32.989440 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:32.989443 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:32.990826 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:32.990857 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:32.991534 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:32.991538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:32.991789 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:32.992352 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:41:32.992374 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:41:32.994756 (+ 2382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:41:32.996665 (+ 1909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:41:32.996672 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:41:32.996692 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:41:32.997078 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:41:32.997436 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:41:32.997439 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:41:32.997441 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:41:32.997651 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:41:32.998074 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:41:32.998082 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:41:32.998391 (+ 309us) client_negotiation.cc:769] Sending connection context
0925 04:41:32.998617 (+ 226us) client_negotiation.cc:240] Negotiation successful
0925 04:41:32.999008 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":144,"threads_started":1}
I20260925 04:41:33.001142 15160 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46403
I20260925 04:41:33.001390 15160 heartbeater.cc:464] Registering TS with master...
I20260925 04:41:33.001941 15160 heartbeater.cc:511] Master 127.0.98.62:46403 requested a full tablet report, sending...
I20260925 04:41:33.003397 14702 ts_manager.cc:194] Registered new tserver with Master: fceaa38dcc7f4a8fae17e2252f431b15 (127.0.98.3:44101)
I20260925 04:41:33.004042 14702 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36697
I20260925 04:41:33.011749 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:41:33.016243 14892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:33.012410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57346 (local address 127.0.98.62:46403)
0925 04:41:33.012557 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 04:41:33.012560 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:41:33.012691 (+ 131us) server_negotiation.cc:407] Connection header received
0925 04:41:33.012874 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:41:33.012877 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:41:33.012997 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:41:33.013095 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:41:33.013271 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:33.013809 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:33.014718 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:33.014965 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:33.015058 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:33.015344 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:41:33.015871 (+ 527us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:41:33.015989 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 04:41:33.016061 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 04:41:33.016124 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 04:41:33.016963 14702 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:57346:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:41:33.020370 14702 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:41:33.028597 14824 tablet_service.cc:1467] Processing CreateTablet for tablet d74a827b023843be9adfc8bec2e2374a (DEFAULT_TABLE table=test-table [id=529dee13508948b7b73a268ff360eb95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:41:33.029336 14824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d74a827b023843be9adfc8bec2e2374a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:41:33.032209 14782 tablet_bootstrap.cc:492] T d74a827b023843be9adfc8bec2e2374a P 7a781f32e0144cbf99f8bbff69c3fdac: Bootstrap starting.
I20260925 04:41:33.033154 14782 tablet_bootstrap.cc:654] T d74a827b023843be9adfc8bec2e2374a P 7a781f32e0144cbf99f8bbff69c3fdac: Neither blocks nor log segments found. Creating new log.
I20260925 04:41:33.033352 15170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:33.028002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44101 (local address 127.0.0.1:40450)
0925 04:41:33.028247 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:33.028259 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:41:33.028338 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:41:33.029213 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:41:33.029216 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:41:33.029237 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:41:33.029325 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:33.029331 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:33.030898 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:33.030901 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:33.032041 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:33.032051 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:33.032903 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:33.032906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:33.032993 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:33.033038 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 04:41:33.033112 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 04:41:33.033201 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":46,"thread_start_us":113,"threads_started":1}
I20260925 04:41:33.033390 15171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:33.028433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40450 (local address 127.0.98.3:44101)
0925 04:41:33.028801 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0925 04:41:33.028812 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 04:41:33.028831 (+ 19us) server_negotiation.cc:407] Connection header received
0925 04:41:33.028900 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:41:33.028905 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:41:33.029072 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:41:33.029496 (+ 424us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:41:33.029654 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:33.030740 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:33.032137 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:33.032764 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:33.032862 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:33.033022 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 04:41:33.033153 (+ 131us) server_negotiation.cc:299] Negotiation successful
0925 04:41:33.033249 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":171,"threads_started":1}
I20260925 04:41:33.033799 14897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:33.027631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41513 (local address 127.0.0.1:50528)
0925 04:41:33.027958 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:33.027973 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:41:33.028049 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:41:33.029086 (+ 1037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:41:33.029090 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:41:33.029109 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:41:33.029222 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:33.029229 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:33.030782 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:33.030790 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:33.032121 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:33.032131 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:33.033401 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:33.033405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:33.033515 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:33.033543 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 04:41:33.033595 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 04:41:33.033672 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":45}
I20260925 04:41:33.034896 14782 tablet_bootstrap.cc:492] T d74a827b023843be9adfc8bec2e2374a P 7a781f32e0144cbf99f8bbff69c3fdac: No bootstrap required, opened a new log
I20260925 04:41:33.034994 14782 ts_tablet_manager.cc:1397] T d74a827b023843be9adfc8bec2e2374a P 7a781f32e0144cbf99f8bbff69c3fdac: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 04:41:33.035394 15169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:33.027980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50528 (local address 127.0.98.2:41513)
0925 04:41:33.028544 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0925 04:41:33.028555 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 04:41:33.028574 (+ 19us) server_negotiation.cc:407] Connection header received
0925 04:41:33.028656 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:41:33.028661 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:41:33.028916 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:41:33.029081 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:41:33.029349 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:33.030569 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:33.032357 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:33.033175 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:33.034910 (+ 1735us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:33.035113 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0925 04:41:33.035167 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 04:41:33.035259 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":129,"threads_started":1}
I20260925 04:41:33.035401 14782 raft_consensus.cc:348] T d74a827b023843be9adfc8bec2e2374a P 7a781f32e0144cbf99f8bbff69c3fdac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08cb310fd578413c950d87af40935374" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41513 } } peers { permanent_uuid: "fceaa38dcc7f4a8fae17e2252f431b15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44101 } } peers { permanent_uuid: "7a781f32e0144cbf99f8bbff69c3fdac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34513 } }
I20260925 04:41:33.035511 14782 raft_consensus.cc:374] T d74a827b023843be9adfc8bec2e2374a P 7a781f32e0144cbf99f8bbff69c3fdac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:41:33.035553 14782 raft_consensus.cc:729] T d74a827b023843be9adfc8bec2e2374a P 7a781f32e0144cbf99f8bbff69c3fdac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a781f32e0144cbf99f8bbff69c3fdac, State: Initialized, Role: FOLLOWER
I20260925 04:41:33.035696 14782 consensus_queue.cc:260] T d74a827b023843be9adfc8bec2e2374a P 7a781f32e0144cbf99f8bbff69c3fdac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08cb310fd578413c950d87af40935374" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41513 } } peers { permanent_uuid: "fceaa38dcc7f4a8fae17e2252f431b15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44101 } } peers { permanent_uuid: "7a781f32e0144cbf99f8bbff69c3fdac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34513 } }
I20260925 04:41:33.035996 14782 ts_tablet_manager.cc:1428] T d74a827b023843be9adfc8bec2e2374a P 7a781f32e0144cbf99f8bbff69c3fdac: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 04:41:33.037094 15094 tablet_service.cc:1467] Processing CreateTablet for tablet d74a827b023843be9adfc8bec2e2374a (DEFAULT_TABLE table=test-table [id=529dee13508948b7b73a268ff360eb95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:41:33.038129 15094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d74a827b023843be9adfc8bec2e2374a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:41:33.038249 14962 tablet_service.cc:1467] Processing CreateTablet for tablet d74a827b023843be9adfc8bec2e2374a (DEFAULT_TABLE table=test-table [id=529dee13508948b7b73a268ff360eb95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:41:33.039077 14962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d74a827b023843be9adfc8bec2e2374a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:41:33.045480 15173 tablet_bootstrap.cc:492] T d74a827b023843be9adfc8bec2e2374a P fceaa38dcc7f4a8fae17e2252f431b15: Bootstrap starting.
I20260925 04:41:33.046581 15174 tablet_bootstrap.cc:492] T d74a827b023843be9adfc8bec2e2374a P 08cb310fd578413c950d87af40935374: Bootstrap starting.
I20260925 04:41:33.048240 15173 tablet_bootstrap.cc:654] T d74a827b023843be9adfc8bec2e2374a P fceaa38dcc7f4a8fae17e2252f431b15: Neither blocks nor log segments found. Creating new log.
I20260925 04:41:33.049223 15174 tablet_bootstrap.cc:654] T d74a827b023843be9adfc8bec2e2374a P 08cb310fd578413c950d87af40935374: Neither blocks nor log segments found. Creating new log.
I20260925 04:41:33.049597 15173 log.cc:824] T d74a827b023843be9adfc8bec2e2374a P fceaa38dcc7f4a8fae17e2252f431b15: Log is configured to *not* fsync() on all Append() calls
I20260925 04:41:33.050067 15174 log.cc:824] T d74a827b023843be9adfc8bec2e2374a P 08cb310fd578413c950d87af40935374: Log is configured to *not* fsync() on all Append() calls
I20260925 04:41:33.052067 15174 tablet_bootstrap.cc:492] T d74a827b023843be9adfc8bec2e2374a P 08cb310fd578413c950d87af40935374: No bootstrap required, opened a new log
I20260925 04:41:33.052068 15173 tablet_bootstrap.cc:492] T d74a827b023843be9adfc8bec2e2374a P fceaa38dcc7f4a8fae17e2252f431b15: No bootstrap required, opened a new log
I20260925 04:41:33.052289 15173 ts_tablet_manager.cc:1397] T d74a827b023843be9adfc8bec2e2374a P fceaa38dcc7f4a8fae17e2252f431b15: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260925 04:41:33.052485 15174 ts_tablet_manager.cc:1397] T d74a827b023843be9adfc8bec2e2374a P 08cb310fd578413c950d87af40935374: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260925 04:41:33.056424 15173 raft_consensus.cc:348] T d74a827b023843be9adfc8bec2e2374a P fceaa38dcc7f4a8fae17e2252f431b15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08cb310fd578413c950d87af40935374" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41513 } } peers { permanent_uuid: "fceaa38dcc7f4a8fae17e2252f431b15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44101 } } peers { permanent_uuid: "7a781f32e0144cbf99f8bbff69c3fdac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34513 } }
I20260925 04:41:33.056424 15174 raft_consensus.cc:348] T d74a827b023843be9adfc8bec2e2374a P 08cb310fd578413c950d87af40935374 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08cb310fd578413c950d87af40935374" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41513 } } peers { permanent_uuid: "fceaa38dcc7f4a8fae17e2252f431b15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44101 } } peers { permanent_uuid: "7a781f32e0144cbf99f8bbff69c3fdac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34513 } }
I20260925 04:41:33.056768 15174 raft_consensus.cc:374] T d74a827b023843be9adfc8bec2e2374a P 08cb310fd578413c950d87af40935374 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:41:33.056768 15173 raft_consensus.cc:374] T d74a827b023843be9adfc8bec2e2374a P fceaa38dcc7f4a8fae17e2252f431b15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:41:33.056823 15174 raft_consensus.cc:729] T d74a827b023843be9adfc8bec2e2374a P 08cb310fd578413c950d87af40935374 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08cb310fd578413c950d87af40935374, State: Initialized, Role: FOLLOWER
I20260925 04:41:33.056823 15173 raft_consensus.cc:729] T d74a827b023843be9adfc8bec2e2374a P fceaa38dcc7f4a8fae17e2252f431b15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fceaa38dcc7f4a8fae17e2252f431b15, State: Initialized, Role: FOLLOWER
I20260925 04:41:33.057328 15173 consensus_queue.cc:260] T d74a827b023843be9adfc8bec2e2374a P fceaa38dcc7f4a8fae17e2252f431b15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08cb310fd578413c950d87af40935374" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41513 } } peers { permanent_uuid: "fceaa38dcc7f4a8fae17e2252f431b15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44101 } } peers { permanent_uuid: "7a781f32e0144cbf99f8bbff69c3fdac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34513 } }
I20260925 04:41:33.057328 15174 consensus_queue.cc:260] T d74a827b023843be9adfc8bec2e2374a P 08cb310fd578413c950d87af40935374 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08cb310fd578413c950d87af40935374" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41513 } } peers { permanent_uuid: "fceaa38dcc7f4a8fae17e2252f431b15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44101 } } peers { permanent_uuid: "7a781f32e0144cbf99f8bbff69c3fdac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34513 } }
I20260925 04:41:33.058480 15173 ts_tablet_manager.cc:1428] T d74a827b023843be9adfc8bec2e2374a P fceaa38dcc7f4a8fae17e2252f431b15: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 04:41:33.058480 15174 ts_tablet_manager.cc:1428] T d74a827b023843be9adfc8bec2e2374a P 08cb310fd578413c950d87af40935374: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260925 04:41:33.058638 15160 heartbeater.cc:503] Master 127.0.98.62:46403 was elected leader, sending a full tablet report...
I20260925 04:41:33.059784 15028 heartbeater.cc:503] Master 127.0.98.62:46403 was elected leader, sending a full tablet report...
W20260925 04:41:33.071977 14891 tablet.cc:2369] T d74a827b023843be9adfc8bec2e2374a P 7a781f32e0144cbf99f8bbff69c3fdac: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 04:41:33.233361 15161 tablet.cc:2369] T d74a827b023843be9adfc8bec2e2374a P fceaa38dcc7f4a8fae17e2252f431b15: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 04:41:33.290076 15029 tablet.cc:2369] T d74a827b023843be9adfc8bec2e2374a P 08cb310fd578413c950d87af40935374: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:41:33.321218 15178 raft_consensus.cc:482] T d74a827b023843be9adfc8bec2e2374a P 08cb310fd578413c950d87af40935374 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:41:33.321432 15178 raft_consensus.cc:504] T d74a827b023843be9adfc8bec2e2374a P 08cb310fd578413c950d87af40935374 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08cb310fd578413c950d87af40935374" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41513 } } peers { permanent_uuid: "fceaa38dcc7f4a8fae17e2252f431b15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44101 } } peers { permanent_uuid: "7a781f32e0144cbf99f8bbff69c3fdac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34513 } }
I20260925 04:41:33.322714 15178 leader_election.cc:288] T d74a827b023843be9adfc8bec2e2374a P 08cb310fd578413c950d87af40935374 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fceaa38dcc7f4a8fae17e2252f431b15 (127.0.98.3:44101), 7a781f32e0144cbf99f8bbff69c3fdac (127.0.98.1:34513)
I20260925 04:41:33.328027 15171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:33.323051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46605 (local address 127.0.98.3:44101)
0925 04:41:33.323270 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 04:41:33.323274 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:41:33.323361 (+ 87us) server_negotiation.cc:407] Connection header received
0925 04:41:33.323506 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:41:33.323509 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:41:33.323558 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:41:33.323677 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:41:33.324185 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:33.324955 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:33.326752 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:33.327425 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:33.327535 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:33.327636 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 04:41:33.327827 (+ 191us) server_negotiation.cc:299] Negotiation successful
0925 04:41:33.327906 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260925 04:41:33.328053 15180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:33.322928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44101 (local address 127.0.98.2:46605)
0925 04:41:33.323248 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:33.323264 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:41:33.323397 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:41:33.323743 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:41:33.323747 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:41:33.323796 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:41:33.324088 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:33.324095 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:33.325079 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:33.325083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:33.326604 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:33.326635 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:33.327552 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:33.327556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:33.327685 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:33.327733 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 04:41:33.327807 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 04:41:33.327887 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":179,"spinlock_wait_cycles":30592,"thread_start_us":88,"threads_started":1}
I20260925 04:41:33.328353 15179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:33.322944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34513 (local address 127.0.98.2:34421)
0925 04:41:33.323335 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:33.323350 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:41:33.323419 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:41:33.323743 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:41:33.323747 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:41:33.323798 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:41:33.324024 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:33.324033 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:33.325079 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:33.325083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:33.326613 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:33.326621 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:33.327681 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:33.327684 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:33.328036 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:33.328057 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 04:41:33.328123 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 04:41:33.328203 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"mutex_wait_us":428,"spinlock_wait_cycles":25088,"thread_start_us":81,"threads_started":1}
I20260925 04:41:33.328769 15181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:33.323105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34421 (local address 127.0.98.1:34513)
0925 04:41:33.323331 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 04:41:33.323335 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:41:33.323415 (+ 80us) server_negotiation.cc:407] Connection header received
0925 04:41:33.323502 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:41:33.323505 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:41:33.323574 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:41:33.323707 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:41:33.324159 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:33.324960 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:33.326889 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:33.327543 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:33.327742 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:33.327915 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 04:41:33.328590 (+ 675us) server_negotiation.cc:299] Negotiation successful
0925 04:41:33.328670 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":79,"threads_started":1}
I20260925 04:41:33.328804 15114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d74a827b023843be9adfc8bec2e2374a" candidate_uuid: "08cb310fd578413c950d87af40935374" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fceaa38dcc7f4a8fae17e2252f431b15" is_pre_election: true
I20260925 04:41:33.329139 15114 raft_consensus.cc:2454] T d74a827b023843be9adfc8bec2e2374a P fceaa38dcc7f4a8fae17e2252f431b15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08cb310fd578413c950d87af40935374 in term 0.
I20260925 04:41:33.329552 14844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d74a827b023843be9adfc8bec2e2374a" candidate_uuid: "08cb310fd578413c950d87af40935374" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a781f32e0144cbf99f8bbff69c3fdac" is_pre_election: true
I20260925 04:41:33.329598 14918 leader_election.cc:302] T d74a827b023843be9adfc8bec2e2374a P 08cb310fd578413c950d87af40935374 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08cb310fd578413c950d87af40935374, fceaa38dcc7f4a8fae17e2252f431b15; no voters:
I20260925 04:41:33.329821 14844 raft_consensus.cc:2454] T d74a827b023843be9adfc8bec2e2374a P 7a781f32e0144cbf99f8bbff69c3fdac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08cb310fd578413c950d87af40935374 in term 0.
I20260925 04:41:33.329861 15178 raft_consensus.cc:2781] T d74a827b023843be9adfc8bec2e2374a P 08cb310fd578413c950d87af40935374 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:41:33.329927 15178 raft_consensus.cc:482] T d74a827b023843be9adfc8bec2e2374a P 08cb310fd578413c950d87af40935374 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:41:33.329977 15178 raft_consensus.cc:3037] T d74a827b023843be9adfc8bec2e2374a P 08cb310fd578413c950d87af40935374 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:41:33.331084 15178 raft_consensus.cc:504] T d74a827b023843be9adfc8bec2e2374a P 08cb310fd578413c950d87af40935374 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08cb310fd578413c950d87af40935374" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41513 } } peers { permanent_uuid: "fceaa38dcc7f4a8fae17e2252f431b15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44101 } } peers { permanent_uuid: "7a781f32e0144cbf99f8bbff69c3fdac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34513 } }
I20260925 04:41:33.331442 15178 leader_election.cc:288] T d74a827b023843be9adfc8bec2e2374a P 08cb310fd578413c950d87af40935374 [CANDIDATE]: Term 1 election: Requested vote from peers fceaa38dcc7f4a8fae17e2252f431b15 (127.0.98.3:44101), 7a781f32e0144cbf99f8bbff69c3fdac (127.0.98.1:34513)
I20260925 04:41:33.331811 15114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d74a827b023843be9adfc8bec2e2374a" candidate_uuid: "08cb310fd578413c950d87af40935374" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fceaa38dcc7f4a8fae17e2252f431b15"
I20260925 04:41:33.331844 14844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d74a827b023843be9adfc8bec2e2374a" candidate_uuid: "08cb310fd578413c950d87af40935374" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a781f32e0144cbf99f8bbff69c3fdac"
I20260925 04:41:33.331956 14844 raft_consensus.cc:3037] T d74a827b023843be9adfc8bec2e2374a P 7a781f32e0144cbf99f8bbff69c3fdac [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:41:33.331934 15114 raft_consensus.cc:3037] T d74a827b023843be9adfc8bec2e2374a P fceaa38dcc7f4a8fae17e2252f431b15 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:41:33.333146 15114 raft_consensus.cc:2454] T d74a827b023843be9adfc8bec2e2374a P fceaa38dcc7f4a8fae17e2252f431b15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08cb310fd578413c950d87af40935374 in term 1.
I20260925 04:41:33.333144 14844 raft_consensus.cc:2454] T d74a827b023843be9adfc8bec2e2374a P 7a781f32e0144cbf99f8bbff69c3fdac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08cb310fd578413c950d87af40935374 in term 1.
I20260925 04:41:33.333443 14918 leader_election.cc:302] T d74a827b023843be9adfc8bec2e2374a P 08cb310fd578413c950d87af40935374 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08cb310fd578413c950d87af40935374, fceaa38dcc7f4a8fae17e2252f431b15; no voters:
I20260925 04:41:33.333617 15178 raft_consensus.cc:2781] T d74a827b023843be9adfc8bec2e2374a P 08cb310fd578413c950d87af40935374 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:41:33.333844 15178 raft_consensus.cc:686] T d74a827b023843be9adfc8bec2e2374a P 08cb310fd578413c950d87af40935374 [term 1 LEADER]: Becoming Leader. State: Replica: 08cb310fd578413c950d87af40935374, State: Running, Role: LEADER
I20260925 04:41:33.334196 15178 consensus_queue.cc:237] T d74a827b023843be9adfc8bec2e2374a P 08cb310fd578413c950d87af40935374 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08cb310fd578413c950d87af40935374" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41513 } } peers { permanent_uuid: "fceaa38dcc7f4a8fae17e2252f431b15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44101 } } peers { permanent_uuid: "7a781f32e0144cbf99f8bbff69c3fdac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34513 } }
I20260925 04:41:33.337831 14701 catalog_manager.cc:5495] T d74a827b023843be9adfc8bec2e2374a P 08cb310fd578413c950d87af40935374 reported cstate change: term changed from 0 to 1, leader changed from <none> to 08cb310fd578413c950d87af40935374 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "08cb310fd578413c950d87af40935374" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08cb310fd578413c950d87af40935374" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fceaa38dcc7f4a8fae17e2252f431b15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a781f32e0144cbf99f8bbff69c3fdac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34513 } health_report { overall_health: UNKNOWN } } }
I20260925 04:41:33.407476 15169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:33.403760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50538 (local address 127.0.98.2:41513)
0925 04:41:33.403897 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 04:41:33.403900 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:41:33.403914 (+ 14us) server_negotiation.cc:407] Connection header received
0925 04:41:33.404106 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:41:33.404108 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:41:33.404176 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:41:33.404272 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:41:33.404535 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:33.405047 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:33.405938 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:33.406232 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:33.406323 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:33.406630 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:41:33.407062 (+ 432us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:41:33.407151 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 04:41:33.407272 (+ 121us) server_negotiation.cc:299] Negotiation successful
0925 04:41:33.407331 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 04:41:33.413599 15114 raft_consensus.cc:1261] T d74a827b023843be9adfc8bec2e2374a P fceaa38dcc7f4a8fae17e2252f431b15 [term 1 FOLLOWER]: Refusing update from remote peer 08cb310fd578413c950d87af40935374: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:41:33.413591 14844 raft_consensus.cc:1261] T d74a827b023843be9adfc8bec2e2374a P 7a781f32e0144cbf99f8bbff69c3fdac [term 1 FOLLOWER]: Refusing update from remote peer 08cb310fd578413c950d87af40935374: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:41:33.414263 15182 consensus_queue.cc:1035] T d74a827b023843be9adfc8bec2e2374a P 08cb310fd578413c950d87af40935374 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fceaa38dcc7f4a8fae17e2252f431b15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:41:33.414498 15178 consensus_queue.cc:1035] T d74a827b023843be9adfc8bec2e2374a P 08cb310fd578413c950d87af40935374 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a781f32e0144cbf99f8bbff69c3fdac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:41:33.419699 15184 mvcc.cc:205] Tried to move back new op lower bound from 7333115057814310912 to 7333115057499750400. Current Snapshot: MvccSnapshot[applied={T|T < 7333115057814310912}]
I20260925 04:41:33.429100 14702 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:57346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:41:33.429314 14702 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:57346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:41:33.432044 14702 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5ad9c8d894c349b58d60ea2e39f3d704: Sending DeleteTablet for 3 replicas of tablet d74a827b023843be9adfc8bec2e2374a
I20260925 04:41:33.432854 14962 tablet_service.cc:1514] Processing DeleteTablet for tablet d74a827b023843be9adfc8bec2e2374a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:41:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50528
I20260925 04:41:33.432904 14824 tablet_service.cc:1514] Processing DeleteTablet for tablet d74a827b023843be9adfc8bec2e2374a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:41:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45106
I20260925 04:41:33.432984 15094 tablet_service.cc:1514] Processing DeleteTablet for tablet d74a827b023843be9adfc8bec2e2374a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:41:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40450
I20260925 04:41:33.433398 15195 tablet_replica.cc:330] T d74a827b023843be9adfc8bec2e2374a P 08cb310fd578413c950d87af40935374: stopping tablet replica
I20260925 04:41:33.433524 15196 tablet_replica.cc:330] T d74a827b023843be9adfc8bec2e2374a P fceaa38dcc7f4a8fae17e2252f431b15: stopping tablet replica
I20260925 04:41:33.433686 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 14740
I20260925 04:41:33.433763 15195 raft_consensus.cc:2229] T d74a827b023843be9adfc8bec2e2374a P 08cb310fd578413c950d87af40935374 [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:41:33.434171 15196 raft_consensus.cc:2229] T d74a827b023843be9adfc8bec2e2374a P fceaa38dcc7f4a8fae17e2252f431b15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:41:33.434553 15195 raft_consensus.cc:2258] T d74a827b023843be9adfc8bec2e2374a P 08cb310fd578413c950d87af40935374 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:41:33.434687 15196 raft_consensus.cc:2258] T d74a827b023843be9adfc8bec2e2374a P fceaa38dcc7f4a8fae17e2252f431b15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:41:33.436336 15195 ts_tablet_manager.cc:1905] T d74a827b023843be9adfc8bec2e2374a P 08cb310fd578413c950d87af40935374: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:41:33.436349 15196 ts_tablet_manager.cc:1905] T d74a827b023843be9adfc8bec2e2374a P fceaa38dcc7f4a8fae17e2252f431b15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:41:33.439167 15196 ts_tablet_manager.cc:1918] T d74a827b023843be9adfc8bec2e2374a P fceaa38dcc7f4a8fae17e2252f431b15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:41:33.439217 15195 ts_tablet_manager.cc:1918] T d74a827b023843be9adfc8bec2e2374a P 08cb310fd578413c950d87af40935374: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:41:33.439322 15195 log.cc:1196] T d74a827b023843be9adfc8bec2e2374a P 08cb310fd578413c950d87af40935374: Deleting WAL directory at /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-1/wal/wals/d74a827b023843be9adfc8bec2e2374a
I20260925 04:41:33.439457 15196 log.cc:1196] T d74a827b023843be9adfc8bec2e2374a P fceaa38dcc7f4a8fae17e2252f431b15: Deleting WAL directory at /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-2/wal/wals/d74a827b023843be9adfc8bec2e2374a
I20260925 04:41:33.439697 15195 ts_tablet_manager.cc:1939] T d74a827b023843be9adfc8bec2e2374a P 08cb310fd578413c950d87af40935374: Deleting consensus metadata
I20260925 04:41:33.439985 15196 ts_tablet_manager.cc:1939] T d74a827b023843be9adfc8bec2e2374a P fceaa38dcc7f4a8fae17e2252f431b15: Deleting consensus metadata
I20260925 04:41:33.441062 14686 catalog_manager.cc:4841] TS 08cb310fd578413c950d87af40935374 (127.0.98.2:41513): tablet d74a827b023843be9adfc8bec2e2374a (table test-table [id=529dee13508948b7b73a268ff360eb95]) successfully deleted
I20260925 04:41:33.441201 14689 catalog_manager.cc:4841] TS fceaa38dcc7f4a8fae17e2252f431b15 (127.0.98.3:44101): tablet d74a827b023843be9adfc8bec2e2374a (table test-table [id=529dee13508948b7b73a268ff360eb95]) successfully deleted
W20260925 04:41:33.441802 14687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34513 (error 108)
I20260925 04:41:33.442366 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 14895
I20260925 04:41:33.443094 14897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:33.442566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34513 (local address 127.0.0.1:45114)
0925 04:41:33.442931 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:33.443004 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274}
W20260925 04:41:33.443334 14687 catalog_manager.cc:4584] TS 7a781f32e0144cbf99f8bbff69c3fdac (127.0.98.1:34513): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d74a827b023843be9adfc8bec2e2374a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34513: connect: Connection refused (error 111)
I20260925 04:41:33.448040 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 15031
I20260925 04:41:33.453223 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 14669
I20260925 04:41:33.458962 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46403
--webserver_interface=127.0.98.62
--webserver_port=34573
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46403
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:41:33.576323 15198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:41:33.576743 15198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:41:33.576833 15198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:41:33.581435 15198 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:41:33.581528 15198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:41:33.581557 15198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:41:33.581681 15198 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:41:33.581730 15198 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:41:33.587653 15198 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46403
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=34573
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:41:33.588797 15198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:41:33.589841 15198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:41:33.598889 15203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:41:33.598928 15198 server_base.cc:1034] running on GCE node
W20260925 04:41:33.598896 15204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:41:33.598886 15206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:41:33.599735 15198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:41:33.600911 15198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:41:33.602080 15198 hybrid_clock.cc:648] HybridClock initialized: now 1790311293602056 us; error 43 us; skew 500 ppm
Sep 25 04:41:33 dist-test-slave-dc95 krb5kdc[14118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:41:33.605057 15198 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 04:41:33.606045 15198 webserver.cc:460] Webserver started at http://127.0.98.62:34573/ using document root <none> and password file <none>
I20260925 04:41:33.606617 15198 fs_manager.cc:359] Metadata directory not provided
I20260925 04:41:33.606680 15198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:41:33.610743 15198 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:41:33.613117 15213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:41:33.614437 15198 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 04:41:33.614557 15198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/master-0/wal
uuid: "5ad9c8d894c349b58d60ea2e39f3d704"
format_stamp: "Formatted at 2026-09-25 04:41:30 on dist-test-slave-dc95"
server_key: "1a99b4b373f3cbe241fa387b7cee2418"
server_key_iv: "a44e137621847188cdc1c3967ae690c3"
server_key_version: "encryptionkey@0"
I20260925 04:41:33.615142 15198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:41:33.630961 15198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:41:33.634768 15198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:41:33.634959 15198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:41:33.642338 15198 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46403
I20260925 04:41:33.642616 15265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46403 every 8 connection(s)
I20260925 04:41:33.643502 15198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
I20260925 04:41:33.645529 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 15198
I20260925 04:41:33.646139 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:34513
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:46403
--webserver_port=35049
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:41:33.648866 15266 sys_catalog.cc:263] Verifying existing consensus state
I20260925 04:41:33.650681 15266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ad9c8d894c349b58d60ea2e39f3d704: Bootstrap starting.
I20260925 04:41:33.668035 15266 log.cc:824] T 00000000000000000000000000000000 P 5ad9c8d894c349b58d60ea2e39f3d704: Log is configured to *not* fsync() on all Append() calls
I20260925 04:41:33.679418 15266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ad9c8d894c349b58d60ea2e39f3d704: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 04:41:33.679848 15266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ad9c8d894c349b58d60ea2e39f3d704: Bootstrap complete.
I20260925 04:41:33.683890 15266 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ad9c8d894c349b58d60ea2e39f3d704 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad9c8d894c349b58d60ea2e39f3d704" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46403 } }
I20260925 04:41:33.684325 15266 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ad9c8d894c349b58d60ea2e39f3d704 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ad9c8d894c349b58d60ea2e39f3d704, State: Initialized, Role: FOLLOWER
I20260925 04:41:33.684813 15266 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ad9c8d894c349b58d60ea2e39f3d704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad9c8d894c349b58d60ea2e39f3d704" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46403 } }
I20260925 04:41:33.684930 15266 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ad9c8d894c349b58d60ea2e39f3d704 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:41:33.684996 15266 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ad9c8d894c349b58d60ea2e39f3d704 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:41:33.685087 15266 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ad9c8d894c349b58d60ea2e39f3d704 [term 2 FOLLOWER]: Advancing to term 3
I20260925 04:41:33.687121 15266 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ad9c8d894c349b58d60ea2e39f3d704 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad9c8d894c349b58d60ea2e39f3d704" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46403 } }
I20260925 04:41:33.687474 15266 leader_election.cc:302] T 00000000000000000000000000000000 P 5ad9c8d894c349b58d60ea2e39f3d704 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ad9c8d894c349b58d60ea2e39f3d704; no voters:
I20260925 04:41:33.687810 15266 leader_election.cc:288] T 00000000000000000000000000000000 P 5ad9c8d894c349b58d60ea2e39f3d704 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 04:41:33.687937 15270 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ad9c8d894c349b58d60ea2e39f3d704 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 04:41:33.688150 15270 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ad9c8d894c349b58d60ea2e39f3d704 [term 3 LEADER]: Becoming Leader. State: Replica: 5ad9c8d894c349b58d60ea2e39f3d704, State: Running, Role: LEADER
I20260925 04:41:33.688457 15270 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ad9c8d894c349b58d60ea2e39f3d704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad9c8d894c349b58d60ea2e39f3d704" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46403 } }
I20260925 04:41:33.689144 15266 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ad9c8d894c349b58d60ea2e39f3d704 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:41:33.689962 15272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ad9c8d894c349b58d60ea2e39f3d704 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ad9c8d894c349b58d60ea2e39f3d704. Latest consensus state: current_term: 3 leader_uuid: "5ad9c8d894c349b58d60ea2e39f3d704" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad9c8d894c349b58d60ea2e39f3d704" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46403 } } }
I20260925 04:41:33.690092 15272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ad9c8d894c349b58d60ea2e39f3d704 [sys.catalog]: This master's current role is: LEADER
I20260925 04:41:33.689970 15271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ad9c8d894c349b58d60ea2e39f3d704 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "5ad9c8d894c349b58d60ea2e39f3d704" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad9c8d894c349b58d60ea2e39f3d704" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46403 } } }
I20260925 04:41:33.690397 15271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ad9c8d894c349b58d60ea2e39f3d704 [sys.catalog]: This master's current role is: LEADER
I20260925 04:41:33.690770 15279 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:41:33.696632 15279 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:41:33.697439 15279 catalog_manager.cc:1259] Loaded cluster ID: 1235377a389c4d639f6e15e42f920f1f
I20260925 04:41:33.697506 15279 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260925 04:41:33.698483 15279 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f21f70bc9ad google::LogMessage::Fail() at ??:0
@ 0x7f21f70bf88a google::LogMessage::SendToLog() at ??:0
@ 0x7f21f70bc4de google::LogMessage::Flush() at ??:0
@ 0x7f21f70bda19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f22019fbfdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f22019fcd2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f22019f9579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f2201a3d640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f21f930e3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f21f930ec83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f21f93103f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f21f92ff512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f21fa9436db start_thread at ??:0
@ 0x7f21f5bba61f clone at ??:0
W20260925 04:41:33.780103 15268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:41:33.780401 15268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:41:33.780464 15268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:41:33.785053 15268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:41:33.785197 15268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:41:33.785285 15268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 04:41:33.791181 15268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:34513
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35049
--tserver_master_addrs=127.0.98.62:46403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15268
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:41:33.792441 15268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:41:33.793542 15268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:41:33.803874 15310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:41:33.803985 15307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:41:33.803865 15308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:41:33.804351 15268 server_base.cc:1034] running on GCE node
I20260925 04:41:33.804821 15268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:41:33.805526 15268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:41:33.806730 15268 hybrid_clock.cc:648] HybridClock initialized: now 1790311293806711 us; error 32 us; skew 500 ppm
Sep 25 04:41:33 dist-test-slave-dc95 krb5kdc[14118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:41:33.809742 15268 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 04:41:33.810724 15268 webserver.cc:460] Webserver started at http://127.0.98.1:35049/ using document root <none> and password file <none>
I20260925 04:41:33.811305 15268 fs_manager.cc:359] Metadata directory not provided
I20260925 04:41:33.811367 15268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:41:33.815541 15268 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:41:33.817982 15317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:41:33.819443 15268 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:41:33.819569 15268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-0/wal
uuid: "7a781f32e0144cbf99f8bbff69c3fdac"
format_stamp: "Formatted at 2026-09-25 04:41:31 on dist-test-slave-dc95"
server_key: "6893d856fc402642507b48120ed8ef4a"
server_key_iv: "3424d3d43039b8b6596f7353b3cd293a"
server_key_version: "encryptionkey@0"
I20260925 04:41:33.820230 15268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:41:33.857244 15268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:41:33.861202 15268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:41:33.861408 15268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:41:33.862133 15268 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:41:33.863494 15324 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 04:41:33.869095 15268 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 04:41:33.869242 15268 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260925 04:41:33.869328 15268 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 04:41:33.871171 15268 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 04:41:33.871229 15268 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260925 04:41:33.871578 15324 tablet_bootstrap.cc:492] T d74a827b023843be9adfc8bec2e2374a P 7a781f32e0144cbf99f8bbff69c3fdac: Bootstrap starting.
I20260925 04:41:33.880955 15268 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34513
I20260925 04:41:33.880981 15431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34513 every 8 connection(s)
I20260925 04:41:33.881975 15268 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
I20260925 04:41:33.883914 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 15268
I20260925 04:41:33.884359 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:41513
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:46403
--webserver_port=36097
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:41:33.893548 15324 log.cc:824] T d74a827b023843be9adfc8bec2e2374a P 7a781f32e0144cbf99f8bbff69c3fdac: Log is configured to *not* fsync() on all Append() calls
I20260925 04:41:33.903431 15324 tablet_bootstrap.cc:492] T d74a827b023843be9adfc8bec2e2374a P 7a781f32e0144cbf99f8bbff69c3fdac: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 04:41:33.903915 15324 tablet_bootstrap.cc:492] T d74a827b023843be9adfc8bec2e2374a P 7a781f32e0144cbf99f8bbff69c3fdac: Bootstrap complete.
I20260925 04:41:33.905272 15324 ts_tablet_manager.cc:1397] T d74a827b023843be9adfc8bec2e2374a P 7a781f32e0144cbf99f8bbff69c3fdac: Time spent bootstrapping tablet: real 0.034s user 0.021s sys 0.011s
I20260925 04:41:33.908738 15324 raft_consensus.cc:348] T d74a827b023843be9adfc8bec2e2374a P 7a781f32e0144cbf99f8bbff69c3fdac [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08cb310fd578413c950d87af40935374" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41513 } } peers { permanent_uuid: "fceaa38dcc7f4a8fae17e2252f431b15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44101 } } peers { permanent_uuid: "7a781f32e0144cbf99f8bbff69c3fdac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34513 } }
I20260925 04:41:33.909358 15324 raft_consensus.cc:729] T d74a827b023843be9adfc8bec2e2374a P 7a781f32e0144cbf99f8bbff69c3fdac [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a781f32e0144cbf99f8bbff69c3fdac, State: Initialized, Role: FOLLOWER
I20260925 04:41:33.909924 15324 consensus_queue.cc:260] T d74a827b023843be9adfc8bec2e2374a P 7a781f32e0144cbf99f8bbff69c3fdac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08cb310fd578413c950d87af40935374" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41513 } } peers { permanent_uuid: "fceaa38dcc7f4a8fae17e2252f431b15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44101 } } peers { permanent_uuid: "7a781f32e0144cbf99f8bbff69c3fdac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34513 } }
I20260925 04:41:33.911103 15324 ts_tablet_manager.cc:1428] T d74a827b023843be9adfc8bec2e2374a P 7a781f32e0144cbf99f8bbff69c3fdac: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260925 04:41:33.941828 15434 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:41:33.884220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.1:40931)
0925 04:41:33.884873 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:33.884950 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0925 04:41:33.886090 (+ 1140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:41:33.941032 (+ 54942us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46403: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":170,"threads_started":1}
W20260925 04:41:33.942380 15432 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46403 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:46403: Client connection negotiation failed: client connection to 127.0.98.62:46403: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 04:41:33.943190 15434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:33.942896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.1:57885)
0925 04:41:33.943039 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:33.943119 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260925 04:41:33.943925 15434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:33.943716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.1:43299)
0925 04:41:33.943817 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:33.943867 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 04:41:33.944120 15432 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 04:41:34.004662 15435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:41:34.004951 15435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:41:34.004998 15435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:41:34.009334 15435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:41:34.009431 15435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:41:34.009498 15435 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 04:41:34.015233 15435 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:41513
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36097
--tserver_master_addrs=127.0.98.62:46403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:41:34.016376 15435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:41:34.017400 15435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:41:34.027123 15443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:41:34.027149 15445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:41:34.027189 15442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:41:34.027560 15435 server_base.cc:1034] running on GCE node
I20260925 04:41:34.028010 15435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:41:34.028738 15435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:41:34.029934 15435 hybrid_clock.cc:648] HybridClock initialized: now 1790311294029922 us; error 70 us; skew 500 ppm
Sep 25 04:41:34 dist-test-slave-dc95 krb5kdc[14118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311294, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:41:34.032773 15435 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 04:41:34.033725 15435 webserver.cc:460] Webserver started at http://127.0.98.2:36097/ using document root <none> and password file <none>
I20260925 04:41:34.034269 15435 fs_manager.cc:359] Metadata directory not provided
I20260925 04:41:34.034333 15435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:41:34.038503 15435 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:41:34.041003 15452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:41:34.042474 15435 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 04:41:34.042600 15435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-1/wal
uuid: "08cb310fd578413c950d87af40935374"
format_stamp: "Formatted at 2026-09-25 04:41:31 on dist-test-slave-dc95"
server_key: "10bb2b5d75ba0c0ee3f56bdfe9fa35c0"
server_key_iv: "51cc0695b753f9e04434fb1294aa76f6"
server_key_version: "encryptionkey@0"
I20260925 04:41:34.043181 15435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:41:34.064342 15435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:41:34.067533 15435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:41:34.067739 15435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:41:34.068369 15435 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:41:34.069226 15435 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:41:34.069278 15435 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:41:34.069327 15435 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:41:34.069341 15435 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:41:34.078605 15435 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41513
I20260925 04:41:34.078698 15565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41513 every 8 connection(s)
I20260925 04:41:34.079723 15435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
I20260925 04:41:34.082561 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 15435
I20260925 04:41:34.083032 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:44101
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:46403
--webserver_port=46641
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:41:34.084494 15568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:34.082092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.2:52503)
0925 04:41:34.082708 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:34.083360 (+ 652us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":132,"threads_started":1}
W20260925 04:41:34.085009 15566 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46403 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:46403: Client connection negotiation failed: client connection to 127.0.98.62:46403: connect: Connection refused (error 111)
I20260925 04:41:34.085786 15568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:34.085518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.2:49891)
0925 04:41:34.085661 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:34.085727 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260925 04:41:34.086436 15568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:34.086190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.2:34377)
0925 04:41:34.086337 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:34.086380 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260925 04:41:34.086613 15566 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 04:41:34.198271 15569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:41:34.198572 15569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:41:34.198634 15569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:41:34.203047 15569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:41:34.203140 15569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:41:34.203234 15569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 04:41:34.208906 15569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:44101
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=46641
--tserver_master_addrs=127.0.98.62:46403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:41:34.210057 15569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:41:34.211103 15569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:41:34.220816 15577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:41:34.220806 15575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:41:34.221012 15569 server_base.cc:1034] running on GCE node
W20260925 04:41:34.220806 15574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:41:34.221521 15569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:41:34.222189 15569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:41:34.223374 15569 hybrid_clock.cc:648] HybridClock initialized: now 1790311294223345 us; error 39 us; skew 500 ppm
Sep 25 04:41:34 dist-test-slave-dc95 krb5kdc[14118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311294, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:41:34.226089 15569 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 04:41:34.227041 15569 webserver.cc:460] Webserver started at http://127.0.98.3:46641/ using document root <none> and password file <none>
I20260925 04:41:34.227586 15569 fs_manager.cc:359] Metadata directory not provided
I20260925 04:41:34.227691 15569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:41:34.231796 15569 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:41:34.234148 15584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:41:34.235476 15569 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:41:34.235646 15569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-2/wal
uuid: "fceaa38dcc7f4a8fae17e2252f431b15"
format_stamp: "Formatted at 2026-09-25 04:41:31 on dist-test-slave-dc95"
server_key: "7d91947786c8621a47da59630897f7c3"
server_key_iv: "d0ec3fc0241e0ade903599fcb32e2cdc"
server_key_version: "encryptionkey@0"
I20260925 04:41:34.236225 15569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:41:34.249682 15569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:41:34.253093 15569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:41:34.253289 15569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:41:34.253923 15569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:41:34.254773 15569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:41:34.254822 15569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:41:34.254863 15569 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:41:34.254896 15569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:41:34.263688 15569 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44101
I20260925 04:41:34.263720 15697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44101 every 8 connection(s)
I20260925 04:41:34.264710 15569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
I20260925 04:41:34.268849 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:34.266925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.3:33065)
0925 04:41:34.267481 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:34.268095 (+ 614us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":122,"threads_started":1}
W20260925 04:41:34.269347 15698 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46403 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:46403: Client connection negotiation failed: client connection to 127.0.98.62:46403: connect: Connection refused (error 111)
I20260925 04:41:34.270138 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:34.269871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.3:36049)
0925 04:41:34.270013 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:34.270070 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 04:41:34.270539 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 15569
I20260925 04:41:34.270969 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:34.270676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.3:49449)
0925 04:41:34.270837 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:34.270897 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260925 04:41:34.271206 15698 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 04:41:34.945166 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:34.944730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.1:39125)
0925 04:41:34.944966 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:34.945014 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":91,"threads_started":1}
I20260925 04:41:35.087920 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:35.087307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.2:57625)
0925 04:41:35.087656 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:35.087736 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":94,"threads_started":1}
I20260925 04:41:35.272374 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:35.271912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.3:60497)
0925 04:41:35.272198 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:35.272261 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":74,"threads_started":1}
I20260925 04:41:35.631896 15705 raft_consensus.cc:482] T d74a827b023843be9adfc8bec2e2374a P 7a781f32e0144cbf99f8bbff69c3fdac [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:41:35.632089 15705 raft_consensus.cc:504] T d74a827b023843be9adfc8bec2e2374a P 7a781f32e0144cbf99f8bbff69c3fdac [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08cb310fd578413c950d87af40935374" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41513 } } peers { permanent_uuid: "fceaa38dcc7f4a8fae17e2252f431b15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44101 } } peers { permanent_uuid: "7a781f32e0144cbf99f8bbff69c3fdac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34513 } }
I20260925 04:41:35.633163 15705 leader_election.cc:288] T d74a827b023843be9adfc8bec2e2374a P 7a781f32e0144cbf99f8bbff69c3fdac [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 08cb310fd578413c950d87af40935374 (127.0.98.2:41513), fceaa38dcc7f4a8fae17e2252f431b15 (127.0.98.3:44101)
Sep 25 04:41:35 dist-test-slave-dc95 krb5kdc[14118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 25 04:41:35 dist-test-slave-dc95 krb5kdc[14118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260925 04:41:35.647149 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:35.633652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42621 (local address 127.0.98.3:44101)
0925 04:41:35.634064 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0925 04:41:35.634084 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 04:41:35.634120 (+ 36us) server_negotiation.cc:407] Connection header received
0925 04:41:35.634802 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:41:35.634808 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:41:35.635165 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:41:35.635455 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:41:35.636858 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:35.637819 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:35.639016 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:35.639499 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:35.639707 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:35.642334 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:41:35.642339 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:41:35.642353 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:41:35.642382 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:41:35.644341 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:35.645075 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:35.645081 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:35.645086 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:35.645196 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:35.645456 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:35.645459 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:35.645461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:35.645868 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:41:35.646006 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 04:41:35.646745 (+ 739us) server_negotiation.cc:299] Negotiation successful
0925 04:41:35.646999 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":119,"threads_started":1}
I20260925 04:41:35.647207 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:35.633465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44101 (local address 127.0.98.1:42621)
0925 04:41:35.633773 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:35.633789 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:41:35.633920 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:41:35.635551 (+ 1631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:41:35.635557 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:41:35.636232 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:41:35.636694 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:35.636710 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:35.637963 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:35.637966 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:35.638901 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:35.638911 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:35.639645 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:35.639650 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:35.639864 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:35.640582 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:41:35.640602 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:41:35.642092 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:41:35.644807 (+ 2715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:41:35.644809 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:41:35.644810 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:41:35.644974 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:41:35.645281 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:41:35.645284 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:41:35.645286 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:41:35.645370 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:41:35.646015 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:41:35.646023 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:41:35.646363 (+ 340us) client_negotiation.cc:769] Sending connection context
0925 04:41:35.646607 (+ 244us) client_negotiation.cc:240] Negotiation successful
0925 04:41:35.647025 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":284,"spinlock_wait_cycles":14464,"thread_start_us":66,"threads_started":1}
I20260925 04:41:35.647315 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:35.633476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41513 (local address 127.0.98.1:43041)
0925 04:41:35.633832 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:35.633847 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:41:35.633940 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:41:35.635551 (+ 1611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:41:35.635557 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:41:35.636233 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:41:35.636694 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:35.636710 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:35.637960 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:35.637962 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:35.638898 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:35.638909 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:35.639663 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:35.639666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:35.639907 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:35.641262 (+ 1355us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:41:35.641273 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:41:35.642412 (+ 1139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:41:35.644455 (+ 2043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:41:35.644462 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:41:35.644473 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:41:35.644716 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:41:35.645058 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:41:35.645061 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:41:35.645063 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:41:35.645207 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:41:35.646054 (+ 847us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:41:35.646057 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:41:35.646363 (+ 306us) client_negotiation.cc:769] Sending connection context
0925 04:41:35.646607 (+ 244us) client_negotiation.cc:240] Negotiation successful
0925 04:41:35.646958 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":163,"spinlock_wait_cycles":10368,"thread_start_us":84,"threads_started":1}
I20260925 04:41:35.647998 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:35.633620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43041 (local address 127.0.98.2:41513)
0925 04:41:35.634112 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0925 04:41:35.634123 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 04:41:35.634155 (+ 32us) server_negotiation.cc:407] Connection header received
0925 04:41:35.634802 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:41:35.634808 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:41:35.635166 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:41:35.635455 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:41:35.636858 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:35.637820 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:35.639035 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:35.639523 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:35.639732 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:35.642598 (+ 2866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:41:35.642604 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:41:35.642619 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:41:35.642655 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:41:35.644353 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:35.644809 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:35.644837 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:35.644841 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:35.644959 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:35.645300 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:35.645304 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:35.645305 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:35.645868 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:41:35.646006 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 04:41:35.647493 (+ 1487us) server_negotiation.cc:299] Negotiation successful
0925 04:41:35.647829 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":104,"threads_started":1}
W20260925 04:41:35.650123 15321 leader_election.cc:341] T d74a827b023843be9adfc8bec2e2374a P 7a781f32e0144cbf99f8bbff69c3fdac [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fceaa38dcc7f4a8fae17e2252f431b15 (127.0.98.3:44101): Not found: Tablet not found: d74a827b023843be9adfc8bec2e2374a
W20260925 04:41:35.650354 15318 leader_election.cc:341] T d74a827b023843be9adfc8bec2e2374a P 7a781f32e0144cbf99f8bbff69c3fdac [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 08cb310fd578413c950d87af40935374 (127.0.98.2:41513): Not found: Tablet not found: d74a827b023843be9adfc8bec2e2374a
I20260925 04:41:35.650458 15318 leader_election.cc:302] T d74a827b023843be9adfc8bec2e2374a P 7a781f32e0144cbf99f8bbff69c3fdac [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a781f32e0144cbf99f8bbff69c3fdac; no voters: 08cb310fd578413c950d87af40935374, fceaa38dcc7f4a8fae17e2252f431b15
I20260925 04:41:35.650735 15705 raft_consensus.cc:2726] T d74a827b023843be9adfc8bec2e2374a P 7a781f32e0144cbf99f8bbff69c3fdac [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 04:41:35.946358 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:35.946026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.1:35227)
0925 04:41:35.946188 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:35.946273 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260925 04:41:36.089485 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:36.088961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.2:44451)
0925 04:41:36.089273 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:36.089341 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":98,"threads_started":1}
I20260925 04:41:36.273525 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:36.273125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.3:46807)
0925 04:41:36.273375 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:36.273432 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":89,"threads_started":1}
I20260925 04:41:36.947758 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:36.947296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.1:54113)
0925 04:41:36.947555 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:36.947659 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":75,"threads_started":1}
I20260925 04:41:37.091009 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:37.090521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.2:52721)
0925 04:41:37.090839 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:37.090908 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":86,"threads_started":1}
I20260925 04:41:37.274827 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:37.274366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.3:47337)
0925 04:41:37.274657 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:37.274718 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":94,"threads_started":1}
I20260925 04:41:37.844769 15715 raft_consensus.cc:482] T d74a827b023843be9adfc8bec2e2374a P 7a781f32e0144cbf99f8bbff69c3fdac [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:41:37.844910 15715 raft_consensus.cc:504] T d74a827b023843be9adfc8bec2e2374a P 7a781f32e0144cbf99f8bbff69c3fdac [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08cb310fd578413c950d87af40935374" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41513 } } peers { permanent_uuid: "fceaa38dcc7f4a8fae17e2252f431b15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44101 } } peers { permanent_uuid: "7a781f32e0144cbf99f8bbff69c3fdac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34513 } }
I20260925 04:41:37.845287 15715 leader_election.cc:288] T d74a827b023843be9adfc8bec2e2374a P 7a781f32e0144cbf99f8bbff69c3fdac [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 08cb310fd578413c950d87af40935374 (127.0.98.2:41513), fceaa38dcc7f4a8fae17e2252f431b15 (127.0.98.3:44101)
W20260925 04:41:37.846170 15318 leader_election.cc:341] T d74a827b023843be9adfc8bec2e2374a P 7a781f32e0144cbf99f8bbff69c3fdac [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 08cb310fd578413c950d87af40935374 (127.0.98.2:41513): Not found: Tablet not found: d74a827b023843be9adfc8bec2e2374a
W20260925 04:41:37.846302 15321 leader_election.cc:341] T d74a827b023843be9adfc8bec2e2374a P 7a781f32e0144cbf99f8bbff69c3fdac [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fceaa38dcc7f4a8fae17e2252f431b15 (127.0.98.3:44101): Not found: Tablet not found: d74a827b023843be9adfc8bec2e2374a
I20260925 04:41:37.846369 15321 leader_election.cc:302] T d74a827b023843be9adfc8bec2e2374a P 7a781f32e0144cbf99f8bbff69c3fdac [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a781f32e0144cbf99f8bbff69c3fdac; no voters: 08cb310fd578413c950d87af40935374, fceaa38dcc7f4a8fae17e2252f431b15
I20260925 04:41:37.846539 15715 raft_consensus.cc:2726] T d74a827b023843be9adfc8bec2e2374a P 7a781f32e0144cbf99f8bbff69c3fdac [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 04:41:37.948969 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:37.948519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.1:43849)
0925 04:41:37.948815 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:37.948874 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":79,"threads_started":1}
I20260925 04:41:38.092432 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:38.091982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.2:55267)
0925 04:41:38.092265 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:38.092333 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":79,"threads_started":1}
I20260925 04:41:38.276150 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:38.275742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.3:42671)
0925 04:41:38.276002 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:38.276056 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":87,"threads_started":1}
I20260925 04:41:38.950299 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:38.949847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.1:39349)
0925 04:41:38.950144 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:38.950202 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":82,"threads_started":1}
I20260925 04:41:39.093683 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:39.093254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.2:53197)
0925 04:41:39.093516 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:39.093581 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":99,"threads_started":1}
I20260925 04:41:39.277558 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:39.277073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.3:49893)
0925 04:41:39.277397 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:39.277461 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":152,"threads_started":1}
I20260925 04:41:39.951565 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:39.951114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.1:59145)
0925 04:41:39.951400 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:39.951471 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":71,"threads_started":1}
I20260925 04:41:40.095003 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:40.094547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.2:36113)
0925 04:41:40.094834 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:40.094905 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":79,"threads_started":1}
I20260925 04:41:40.119920 15724 raft_consensus.cc:482] T d74a827b023843be9adfc8bec2e2374a P 7a781f32e0144cbf99f8bbff69c3fdac [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:41:40.120082 15724 raft_consensus.cc:504] T d74a827b023843be9adfc8bec2e2374a P 7a781f32e0144cbf99f8bbff69c3fdac [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08cb310fd578413c950d87af40935374" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41513 } } peers { permanent_uuid: "fceaa38dcc7f4a8fae17e2252f431b15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44101 } } peers { permanent_uuid: "7a781f32e0144cbf99f8bbff69c3fdac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34513 } }
I20260925 04:41:40.120474 15724 leader_election.cc:288] T d74a827b023843be9adfc8bec2e2374a P 7a781f32e0144cbf99f8bbff69c3fdac [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 08cb310fd578413c950d87af40935374 (127.0.98.2:41513), fceaa38dcc7f4a8fae17e2252f431b15 (127.0.98.3:44101)
W20260925 04:41:40.121292 15321 leader_election.cc:341] T d74a827b023843be9adfc8bec2e2374a P 7a781f32e0144cbf99f8bbff69c3fdac [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fceaa38dcc7f4a8fae17e2252f431b15 (127.0.98.3:44101): Not found: Tablet not found: d74a827b023843be9adfc8bec2e2374a
W20260925 04:41:40.121430 15318 leader_election.cc:341] T d74a827b023843be9adfc8bec2e2374a P 7a781f32e0144cbf99f8bbff69c3fdac [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 08cb310fd578413c950d87af40935374 (127.0.98.2:41513): Not found: Tablet not found: d74a827b023843be9adfc8bec2e2374a
I20260925 04:41:40.121498 15318 leader_election.cc:302] T d74a827b023843be9adfc8bec2e2374a P 7a781f32e0144cbf99f8bbff69c3fdac [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a781f32e0144cbf99f8bbff69c3fdac; no voters: 08cb310fd578413c950d87af40935374, fceaa38dcc7f4a8fae17e2252f431b15
I20260925 04:41:40.121665 15724 raft_consensus.cc:2726] T d74a827b023843be9adfc8bec2e2374a P 7a781f32e0144cbf99f8bbff69c3fdac [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 04:41:40.278942 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:40.278486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.3:38823)
0925 04:41:40.278775 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:40.278838 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":93,"threads_started":1}
I20260925 04:41:40.952943 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:40.952455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.1:49109)
0925 04:41:40.952772 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:40.952840 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":80,"threads_started":1}
I20260925 04:41:41.096479 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:41.095958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.2:35729)
0925 04:41:41.096299 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:41.096370 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":77,"threads_started":1}
I20260925 04:41:41.280236 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:41.279754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.3:41909)
0925 04:41:41.280074 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:41.280139 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":72,"threads_started":1}
I20260925 04:41:41.954185 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:41.953698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.1:51815)
0925 04:41:41.953979 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:41.954065 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":77,"threads_started":1}
I20260925 04:41:42.097914 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:42.097392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.2:35241)
0925 04:41:42.097734 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:42.097807 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":89,"threads_started":1}
I20260925 04:41:42.281626 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:42.281176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.3:39307)
0925 04:41:42.281467 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:42.281531 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":83,"threads_started":1}
I20260925 04:41:42.949306 15732 raft_consensus.cc:482] T d74a827b023843be9adfc8bec2e2374a P 7a781f32e0144cbf99f8bbff69c3fdac [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:41:42.949445 15732 raft_consensus.cc:504] T d74a827b023843be9adfc8bec2e2374a P 7a781f32e0144cbf99f8bbff69c3fdac [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08cb310fd578413c950d87af40935374" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41513 } } peers { permanent_uuid: "fceaa38dcc7f4a8fae17e2252f431b15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44101 } } peers { permanent_uuid: "7a781f32e0144cbf99f8bbff69c3fdac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34513 } }
I20260925 04:41:42.949826 15732 leader_election.cc:288] T d74a827b023843be9adfc8bec2e2374a P 7a781f32e0144cbf99f8bbff69c3fdac [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 08cb310fd578413c950d87af40935374 (127.0.98.2:41513), fceaa38dcc7f4a8fae17e2252f431b15 (127.0.98.3:44101)
W20260925 04:41:42.950599 15318 leader_election.cc:341] T d74a827b023843be9adfc8bec2e2374a P 7a781f32e0144cbf99f8bbff69c3fdac [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 08cb310fd578413c950d87af40935374 (127.0.98.2:41513): Not found: Tablet not found: d74a827b023843be9adfc8bec2e2374a
W20260925 04:41:42.950714 15321 leader_election.cc:341] T d74a827b023843be9adfc8bec2e2374a P 7a781f32e0144cbf99f8bbff69c3fdac [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fceaa38dcc7f4a8fae17e2252f431b15 (127.0.98.3:44101): Not found: Tablet not found: d74a827b023843be9adfc8bec2e2374a
I20260925 04:41:42.950796 15321 leader_election.cc:302] T d74a827b023843be9adfc8bec2e2374a P 7a781f32e0144cbf99f8bbff69c3fdac [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a781f32e0144cbf99f8bbff69c3fdac; no voters: 08cb310fd578413c950d87af40935374, fceaa38dcc7f4a8fae17e2252f431b15
I20260925 04:41:42.950928 15732 raft_consensus.cc:2726] T d74a827b023843be9adfc8bec2e2374a P 7a781f32e0144cbf99f8bbff69c3fdac [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 04:41:42.955459 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:42.954991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.1:49767)
0925 04:41:42.955302 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:42.955365 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":94,"threads_started":1}
I20260925 04:41:43.099490 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:43.098980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.2:36279)
0925 04:41:43.099297 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:43.099386 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":93,"threads_started":1}
I20260925 04:41:43.282964 15735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:43.282472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.3:39205)
0925 04:41:43.282796 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:43.282863 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":86,"threads_started":1}
I20260925 04:41:43.956825 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:43.956373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.1:57509)
0925 04:41:43.956659 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:43.956726 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":76,"threads_started":1}
I20260925 04:41:44.100749 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:44.100262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.2:45863)
0925 04:41:44.100587 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:44.100651 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":117,"threads_started":1}
I20260925 04:41:44.284476 15738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:44.283992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.3:44331)
0925 04:41:44.284306 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:44.284370 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":99,"threads_started":1}
I20260925 04:41:44.527752 15739 raft_consensus.cc:482] T d74a827b023843be9adfc8bec2e2374a P 7a781f32e0144cbf99f8bbff69c3fdac [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:41:44.527906 15739 raft_consensus.cc:504] T d74a827b023843be9adfc8bec2e2374a P 7a781f32e0144cbf99f8bbff69c3fdac [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08cb310fd578413c950d87af40935374" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41513 } } peers { permanent_uuid: "fceaa38dcc7f4a8fae17e2252f431b15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44101 } } peers { permanent_uuid: "7a781f32e0144cbf99f8bbff69c3fdac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34513 } }
I20260925 04:41:44.528352 15739 leader_election.cc:288] T d74a827b023843be9adfc8bec2e2374a P 7a781f32e0144cbf99f8bbff69c3fdac [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 08cb310fd578413c950d87af40935374 (127.0.98.2:41513), fceaa38dcc7f4a8fae17e2252f431b15 (127.0.98.3:44101)
W20260925 04:41:44.529053 15318 leader_election.cc:341] T d74a827b023843be9adfc8bec2e2374a P 7a781f32e0144cbf99f8bbff69c3fdac [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 08cb310fd578413c950d87af40935374 (127.0.98.2:41513): Not found: Tablet not found: d74a827b023843be9adfc8bec2e2374a
W20260925 04:41:44.529158 15321 leader_election.cc:341] T d74a827b023843be9adfc8bec2e2374a P 7a781f32e0144cbf99f8bbff69c3fdac [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fceaa38dcc7f4a8fae17e2252f431b15 (127.0.98.3:44101): Not found: Tablet not found: d74a827b023843be9adfc8bec2e2374a
I20260925 04:41:44.529251 15321 leader_election.cc:302] T d74a827b023843be9adfc8bec2e2374a P 7a781f32e0144cbf99f8bbff69c3fdac [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a781f32e0144cbf99f8bbff69c3fdac; no voters: 08cb310fd578413c950d87af40935374, fceaa38dcc7f4a8fae17e2252f431b15
I20260925 04:41:44.529387 15739 raft_consensus.cc:2726] T d74a827b023843be9adfc8bec2e2374a P 7a781f32e0144cbf99f8bbff69c3fdac [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 04:41:44.958258 15740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:44.957784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.1:38407)
0925 04:41:44.958070 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:44.958138 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":90,"threads_started":1}
I20260925 04:41:45.101954 15741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:45.101563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.2:51389)
0925 04:41:45.101809 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:45.101861 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":73,"threads_started":1}
I20260925 04:41:45.285923 15742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:45.285403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.3:53127)
0925 04:41:45.285758 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:45.285824 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":84,"threads_started":1}
I20260925 04:41:45.959678 15743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:45.959189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.1:49569)
0925 04:41:45.959467 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:45.959532 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":79,"threads_started":1}
I20260925 04:41:46.103219 15744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:46.102766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.2:35575)
0925 04:41:46.103039 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:46.103106 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":74,"threads_started":1}
I20260925 04:41:46.287286 15745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:46.286828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.3:44307)
0925 04:41:46.287105 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:46.287189 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":77,"threads_started":1}
I20260925 04:41:46.960994 15746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:46.960491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.1:60053)
0925 04:41:46.960807 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:46.960875 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":92,"threads_started":1}
I20260925 04:41:47.104453 15747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:47.104033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.2:46435)
0925 04:41:47.104293 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:47.104356 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":87,"threads_started":1}
I20260925 04:41:47.288496 15748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:47.288087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.3:54401)
0925 04:41:47.288344 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:47.288400 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":72,"threads_started":1}
I20260925 04:41:47.962285 15749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:47.961835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.1:33741)
0925 04:41:47.962124 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:47.962183 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":91,"threads_started":1}
I20260925 04:41:48.105830 15750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:48.105282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.2:57785)
0925 04:41:48.105629 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:48.105707 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":119,"threads_started":1}
I20260925 04:41:48.289897 15751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:48.289409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.3:37979)
0925 04:41:48.289718 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:48.289794 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":128,"threads_started":1}
I20260925 04:41:48.963894 15752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:48.963364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.1:42955)
0925 04:41:48.963725 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:48.963791 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":115,"threads_started":1}
I20260925 04:41:49.107254 15753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:49.106822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.2:59689)
0925 04:41:49.107091 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:49.107157 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":84,"threads_started":1}
I20260925 04:41:49.277786 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 15268
I20260925 04:41:49.283154 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 15435
I20260925 04:41:49.287945 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 15569
2026-09-25T04:41:49Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19705 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:41:49 dist-test-slave-dc95 krb5kdc[15756](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:41:49 dist-test-slave-dc95 krb5kdc[15756](info): set up 2 sockets
Sep 25 04:41:49 dist-test-slave-dc95 krb5kdc[15756](info): commencing operation
krb5kdc: starting...
W20260925 04:41:50.251567 392 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/krb5kdc/test-user.keytab.
Sep 25 04:41:50 dist-test-slave-dc95 krb5kdc[15756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:41:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:41:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 04:41:50.376040 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38799
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38799
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:41:50.491375 15769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:41:50.491719 15769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:41:50.491772 15769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:41:50.496086 15769 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 04:41:50.496163 15769 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 04:41:50.496178 15769 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 04:41:50.496191 15769 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:41:50.496208 15769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:41:50.496227 15769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:41:50.496240 15769 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:41:50.496254 15769 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:41:50.502079 15769 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15769
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:41:50.503228 15769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:41:50.504323 15769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:41:50.513335 15775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:41:50.513399 15774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:41:50.513335 15777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:41:50.513622 15769 server_base.cc:1034] running on GCE node
I20260925 04:41:50.514364 15769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:41:50.515793 15769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:41:50.516992 15769 hybrid_clock.cc:648] HybridClock initialized: now 1790311310516966 us; error 44 us; skew 500 ppm
Sep 25 04:41:50 dist-test-slave-dc95 krb5kdc[15756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311310, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:41:50.520021 15769 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 04:41:50.521106 15769 webserver.cc:460] Webserver started at http://127.0.98.62:43981/ using document root <none> and password file <none>
I20260925 04:41:50.521667 15769 fs_manager.cc:359] Metadata directory not provided
I20260925 04:41:50.521733 15769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:41:50.521926 15769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:41:50.524147 15769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/master-0/data/instance:
uuid: "7fc2639fe7ca454c8de7a2cfce0e63c3"
format_stamp: "Formatted at 2026-09-25 04:41:50 on dist-test-slave-dc95"
server_key: "7541e435ee78e02d5809b5333f7ccd0c"
server_key_iv: "8dfe44d99c78a72e7614d359866b216c"
server_key_version: "encryptionkey@0"
I20260925 04:41:50.524729 15769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/master-0/wal/instance:
uuid: "7fc2639fe7ca454c8de7a2cfce0e63c3"
format_stamp: "Formatted at 2026-09-25 04:41:50 on dist-test-slave-dc95"
server_key: "7541e435ee78e02d5809b5333f7ccd0c"
server_key_iv: "8dfe44d99c78a72e7614d359866b216c"
server_key_version: "encryptionkey@0"
I20260925 04:41:50.528776 15769 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:41:50.531494 15784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:41:50.533008 15769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:41:50.533175 15769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/master-0/wal
uuid: "7fc2639fe7ca454c8de7a2cfce0e63c3"
format_stamp: "Formatted at 2026-09-25 04:41:50 on dist-test-slave-dc95"
server_key: "7541e435ee78e02d5809b5333f7ccd0c"
server_key_iv: "8dfe44d99c78a72e7614d359866b216c"
server_key_version: "encryptionkey@0"
I20260925 04:41:50.533298 15769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:41:50.548381 15769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:41:50.551702 15769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:41:50.551894 15769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:41:50.558943 15769 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38799
I20260925 04:41:50.559247 15836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38799 every 8 connection(s)
I20260925 04:41:50.559988 15769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
I20260925 04:41:50.562819 15837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:41:50.563092 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 15769
I20260925 04:41:50.563198 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/master-0/wal/instance
I20260925 04:41:50.563457 392 external_mini_cluster.cc:1428] Setting key 5f6bce1fc452ca0772239f191556e726
I20260925 04:41:50.569881 15837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fc2639fe7ca454c8de7a2cfce0e63c3: Bootstrap starting.
Sep 25 04:41:50 dist-test-slave-dc95 krb5kdc[15756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:41:50.572780 15837 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7fc2639fe7ca454c8de7a2cfce0e63c3: Neither blocks nor log segments found. Creating new log.
I20260925 04:41:50.573676 15837 log.cc:824] T 00000000000000000000000000000000 P 7fc2639fe7ca454c8de7a2cfce0e63c3: Log is configured to *not* fsync() on all Append() calls
I20260925 04:41:50.576191 15837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fc2639fe7ca454c8de7a2cfce0e63c3: No bootstrap required, opened a new log
I20260925 04:41:50.578538 15840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:50.564770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36300 (local address 127.0.98.62:38799)
0925 04:41:50.565182 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0925 04:41:50.565197 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 04:41:50.565232 (+ 35us) server_negotiation.cc:407] Connection header received
0925 04:41:50.565978 (+ 746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:41:50.565997 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:41:50.566338 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:41:50.566631 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:41:50.567557 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:50.568669 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:50.569833 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:50.570373 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:50.570497 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:50.573244 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:41:50.573251 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:41:50.573270 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:41:50.573308 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:41:50.575228 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:50.575732 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:50.575738 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:50.575745 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:50.575892 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:50.576169 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:50.576172 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:50.576174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:50.576770 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:41:50.576959 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0925 04:41:50.577226 (+ 267us) server_negotiation.cc:299] Negotiation successful
0925 04:41:50.577534 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":115,"threads_started":1}
I20260925 04:41:50.580457 15837 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7fc2639fe7ca454c8de7a2cfce0e63c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fc2639fe7ca454c8de7a2cfce0e63c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38799 } }
I20260925 04:41:50.580693 15837 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7fc2639fe7ca454c8de7a2cfce0e63c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:41:50.580787 15837 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7fc2639fe7ca454c8de7a2cfce0e63c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fc2639fe7ca454c8de7a2cfce0e63c3, State: Initialized, Role: FOLLOWER
I20260925 04:41:50.581251 15837 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7fc2639fe7ca454c8de7a2cfce0e63c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fc2639fe7ca454c8de7a2cfce0e63c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38799 } }
I20260925 04:41:50.581387 15837 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7fc2639fe7ca454c8de7a2cfce0e63c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:41:50.581444 15837 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7fc2639fe7ca454c8de7a2cfce0e63c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:41:50.581540 15837 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7fc2639fe7ca454c8de7a2cfce0e63c3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:41:50.582618 15837 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7fc2639fe7ca454c8de7a2cfce0e63c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fc2639fe7ca454c8de7a2cfce0e63c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38799 } }
I20260925 04:41:50.582998 15837 leader_election.cc:302] T 00000000000000000000000000000000 P 7fc2639fe7ca454c8de7a2cfce0e63c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7fc2639fe7ca454c8de7a2cfce0e63c3; no voters:
I20260925 04:41:50.583348 15837 leader_election.cc:288] T 00000000000000000000000000000000 P 7fc2639fe7ca454c8de7a2cfce0e63c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:41:50.583446 15842 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7fc2639fe7ca454c8de7a2cfce0e63c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:41:50.583675 15842 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7fc2639fe7ca454c8de7a2cfce0e63c3 [term 1 LEADER]: Becoming Leader. State: Replica: 7fc2639fe7ca454c8de7a2cfce0e63c3, State: Running, Role: LEADER
I20260925 04:41:50.584023 15842 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7fc2639fe7ca454c8de7a2cfce0e63c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fc2639fe7ca454c8de7a2cfce0e63c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38799 } }
I20260925 04:41:50.584542 15837 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7fc2639fe7ca454c8de7a2cfce0e63c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:41:50.585614 15843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fc2639fe7ca454c8de7a2cfce0e63c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7fc2639fe7ca454c8de7a2cfce0e63c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fc2639fe7ca454c8de7a2cfce0e63c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38799 } } }
I20260925 04:41:50.585763 15843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fc2639fe7ca454c8de7a2cfce0e63c3 [sys.catalog]: This master's current role is: LEADER
I20260925 04:41:50.585623 15844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fc2639fe7ca454c8de7a2cfce0e63c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7fc2639fe7ca454c8de7a2cfce0e63c3. Latest consensus state: current_term: 1 leader_uuid: "7fc2639fe7ca454c8de7a2cfce0e63c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fc2639fe7ca454c8de7a2cfce0e63c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38799 } } }
I20260925 04:41:50.585966 15844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fc2639fe7ca454c8de7a2cfce0e63c3 [sys.catalog]: This master's current role is: LEADER
I20260925 04:41:50.586181 15851 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:41:50.589187 15851 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:41:50.595500 15851 catalog_manager.cc:1347] Generated new cluster ID: 04ecd207d1c04912833509ed3e6cccae
I20260925 04:41:50.595625 15851 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:41:50.612145 15851 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:41:50.613152 15851 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:41:50.619326 15851 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7fc2639fe7ca454c8de7a2cfce0e63c3: Generated new TSK 0
I20260925 04:41:50.620096 15851 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 04:41:50.654695 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38799
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:41:50.770829 15863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:41:50.771121 15863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:41:50.771165 15863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:41:50.775363 15863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:41:50.775454 15863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:41:50.775516 15863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 04:41:50.781019 15863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:41:50.782114 15863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:41:50.783110 15863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:41:50.793109 15869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:41:50.793092 15868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:41:50.793170 15863 server_base.cc:1034] running on GCE node
W20260925 04:41:50.793092 15871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:41:50.793845 15863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:41:50.794534 15863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:41:50.795692 15863 hybrid_clock.cc:648] HybridClock initialized: now 1790311310795661 us; error 45 us; skew 500 ppm
Sep 25 04:41:50 dist-test-slave-dc95 krb5kdc[15756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311310, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:41:50.798559 15863 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 04:41:50.799548 15863 webserver.cc:460] Webserver started at http://127.0.98.1:35771/ using document root <none> and password file <none>
I20260925 04:41:50.800175 15863 fs_manager.cc:359] Metadata directory not provided
I20260925 04:41:50.800251 15863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:41:50.800468 15863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:41:50.802639 15863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-0/data/instance:
uuid: "11f96b80805a4844b6650c59f3d524eb"
format_stamp: "Formatted at 2026-09-25 04:41:50 on dist-test-slave-dc95"
server_key: "13b385f8ae8c81800a5364e5d1a66da3"
server_key_iv: "c52952f308cdd5d540d4915cc7bfa80b"
server_key_version: "encryptionkey@0"
I20260925 04:41:50.803247 15863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-0/wal/instance:
uuid: "11f96b80805a4844b6650c59f3d524eb"
format_stamp: "Formatted at 2026-09-25 04:41:50 on dist-test-slave-dc95"
server_key: "13b385f8ae8c81800a5364e5d1a66da3"
server_key_iv: "c52952f308cdd5d540d4915cc7bfa80b"
server_key_version: "encryptionkey@0"
I20260925 04:41:50.807910 15863 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260925 04:41:50.810891 15878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:41:50.812434 15863 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:41:50.812573 15863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-0/wal
uuid: "11f96b80805a4844b6650c59f3d524eb"
format_stamp: "Formatted at 2026-09-25 04:41:50 on dist-test-slave-dc95"
server_key: "13b385f8ae8c81800a5364e5d1a66da3"
server_key_iv: "c52952f308cdd5d540d4915cc7bfa80b"
server_key_version: "encryptionkey@0"
I20260925 04:41:50.812700 15863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:41:50.832373 15863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:41:50.836004 15863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:41:50.836200 15863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:41:50.836870 15863 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:41:50.837749 15863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:41:50.837800 15863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:41:50.837852 15863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:41:50.837879 15863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:41:50.846531 15863 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41203
I20260925 04:41:50.846549 15991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41203 every 8 connection(s)
I20260925 04:41:50.847450 15863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
I20260925 04:41:50.851498 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 15863
I20260925 04:41:50.851652 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-0/wal/instance
I20260925 04:41:50.851918 392 external_mini_cluster.cc:1428] Setting key 3999afd284a6abaa20794ecffb8c4789
Sep 25 04:41:50 dist-test-slave-dc95 krb5kdc[15756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311310, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 04:41:50.862751 15840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:50.849429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59141 (local address 127.0.98.62:38799)
0925 04:41:50.849561 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 04:41:50.849565 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:41:50.850607 (+ 1042us) server_negotiation.cc:407] Connection header received
0925 04:41:50.852003 (+ 1396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:41:50.852007 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:41:50.852067 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:41:50.852140 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:41:50.853387 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:50.853948 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:50.855478 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:50.855802 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:50.855904 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:50.858947 (+ 3043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:41:50.858950 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:41:50.858952 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:41:50.858978 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:41:50.860502 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:50.861149 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:50.861152 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:50.861154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:50.861244 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:50.861593 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:50.861595 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:50.861596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:50.861787 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:41:50.861871 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 04:41:50.862504 (+ 633us) server_negotiation.cc:299] Negotiation successful
0925 04:41:50.862622 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 04:41:50.863708 15994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:50.849851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38799 (local address 127.0.98.1:59141)
0925 04:41:50.850431 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:50.850479 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 04:41:50.851758 (+ 1279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:41:50.852347 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:41:50.852354 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:41:50.852725 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:41:50.853241 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:50.853255 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:50.854077 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:50.854080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:50.855324 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:50.855339 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:50.855969 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:50.855972 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:50.856177 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:50.856877 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:41:50.856898 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:41:50.858716 (+ 1818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:41:50.860646 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:41:50.860653 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:41:50.860673 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:41:50.861027 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:41:50.861328 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:41:50.861331 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:41:50.861333 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:41:50.861511 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:41:50.861883 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:41:50.861888 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:41:50.862269 (+ 381us) client_negotiation.cc:769] Sending connection context
0925 04:41:50.862446 (+ 177us) client_negotiation.cc:240] Negotiation successful
0925 04:41:50.862730 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":127,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260925 04:41:50.865032 15992 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38799
I20260925 04:41:50.865312 15992 heartbeater.cc:464] Registering TS with master...
I20260925 04:41:50.865881 15992 heartbeater.cc:511] Master 127.0.98.62:38799 requested a full tablet report, sending...
I20260925 04:41:50.867491 15801 ts_manager.cc:194] Registered new tserver with Master: 11f96b80805a4844b6650c59f3d524eb (127.0.98.1:41203)
I20260925 04:41:50.868677 15801 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59141
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 04:41:50.882548 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38799
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:41:50.996451 15997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:41:50.996762 15997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:41:50.996807 15997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:41:51.000980 15997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:41:51.001052 15997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:41:51.001114 15997 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 04:41:51.006582 15997 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15997
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:41:51.007733 15997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:41:51.008772 15997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:41:51.018746 16005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:41:51.018814 16002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:41:51.018807 15997 server_base.cc:1034] running on GCE node
W20260925 04:41:51.018756 16003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:41:51.019517 15997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:41:51.020236 15997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:41:51.021425 15997 hybrid_clock.cc:648] HybridClock initialized: now 1790311311021409 us; error 37 us; skew 500 ppm
Sep 25 04:41:51 dist-test-slave-dc95 krb5kdc[15756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:41:51.024219 15997 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 04:41:51.025189 15997 webserver.cc:460] Webserver started at http://127.0.98.2:46187/ using document root <none> and password file <none>
I20260925 04:41:51.025745 15997 fs_manager.cc:359] Metadata directory not provided
I20260925 04:41:51.025821 15997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:41:51.026033 15997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:41:51.028208 15997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-1/data/instance:
uuid: "699d9f5435bf4e2692984cf3bf4acd8c"
format_stamp: "Formatted at 2026-09-25 04:41:51 on dist-test-slave-dc95"
server_key: "5b61352dbd08d2fa97c18084f4038d15"
server_key_iv: "ad94de5435669c9804536b871efed19f"
server_key_version: "encryptionkey@0"
I20260925 04:41:51.028821 15997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-1/wal/instance:
uuid: "699d9f5435bf4e2692984cf3bf4acd8c"
format_stamp: "Formatted at 2026-09-25 04:41:51 on dist-test-slave-dc95"
server_key: "5b61352dbd08d2fa97c18084f4038d15"
server_key_iv: "ad94de5435669c9804536b871efed19f"
server_key_version: "encryptionkey@0"
I20260925 04:41:51.032804 15997 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:41:51.035444 16012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:41:51.036832 15997 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:41:51.037000 15997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-1/wal
uuid: "699d9f5435bf4e2692984cf3bf4acd8c"
format_stamp: "Formatted at 2026-09-25 04:41:51 on dist-test-slave-dc95"
server_key: "5b61352dbd08d2fa97c18084f4038d15"
server_key_iv: "ad94de5435669c9804536b871efed19f"
server_key_version: "encryptionkey@0"
I20260925 04:41:51.037128 15997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:41:51.083419 15997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:41:51.087049 15997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:41:51.087249 15997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:41:51.087949 15997 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:41:51.088819 15997 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:41:51.088868 15997 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:41:51.088924 15997 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:41:51.088953 15997 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:41:51.097786 15997 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45283
I20260925 04:41:51.097831 16125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45283 every 8 connection(s)
I20260925 04:41:51.098767 15997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
I20260925 04:41:51.099664 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 15997
I20260925 04:41:51.099776 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-1/wal/instance
I20260925 04:41:51.100248 392 external_mini_cluster.cc:1428] Setting key 714b1f079722f8d0bdebaaaede29a73f
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Sep 25 04:41:51 dist-test-slave-dc95 krb5kdc[15756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:41:51.118585 15840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:51.100889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48473 (local address 127.0.98.62:38799)
0925 04:41:51.101022 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 04:41:51.101026 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:41:51.103485 (+ 2459us) server_negotiation.cc:407] Connection header received
0925 04:41:51.105070 (+ 1585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:41:51.105073 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:41:51.105123 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:41:51.105191 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:41:51.106941 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:51.107500 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:51.109414 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:51.109712 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:51.109818 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:51.114346 (+ 4528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:41:51.114350 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:41:51.114352 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:41:51.114386 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:41:51.116077 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:51.116837 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:51.116840 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:51.116842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:51.116949 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:51.117378 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:51.117381 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:51.117382 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:51.117587 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:41:51.117679 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 04:41:51.118246 (+ 567us) server_negotiation.cc:299] Negotiation successful
0925 04:41:51.118389 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 04:41:51.119515 16128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:51.101356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38799 (local address 127.0.98.2:48473)
0925 04:41:51.103318 (+ 1962us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:51.103367 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 04:41:51.104785 (+ 1418us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:41:51.105417 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:41:51.105428 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:41:51.105953 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:41:51.106737 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:51.106759 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:51.107678 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:51.107682 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:51.109150 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:51.109164 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:51.109984 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:51.109992 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:51.110217 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:51.110904 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:41:51.110973 (+ 69us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:41:51.114081 (+ 3108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:41:51.116345 (+ 2264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:41:51.116352 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:41:51.116369 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:41:51.116676 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:41:51.117049 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:41:51.117051 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:41:51.117053 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:41:51.117213 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:41:51.117697 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:41:51.117704 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:41:51.117983 (+ 279us) client_negotiation.cc:769] Sending connection context
0925 04:41:51.118161 (+ 178us) client_negotiation.cc:240] Negotiation successful
0925 04:41:51.118551 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1749,"thread_start_us":151,"threads_started":1}
I20260925 04:41:51.120793 16126 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38799
I20260925 04:41:51.121037 16126 heartbeater.cc:464] Registering TS with master...
I20260925 04:41:51.121599 16126 heartbeater.cc:511] Master 127.0.98.62:38799 requested a full tablet report, sending...
I20260925 04:41:51.122723 15801 ts_manager.cc:194] Registered new tserver with Master: 699d9f5435bf4e2692984cf3bf4acd8c (127.0.98.2:45283)
I20260925 04:41:51.123361 15801 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48473
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 04:41:51.131459 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38799
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:41:51.245525 16131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:41:51.245819 16131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:41:51.245863 16131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:41:51.250182 16131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:41:51.250274 16131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:41:51.250339 16131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 04:41:51.256567 16131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16131
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:41:51.257741 16131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:41:51.258769 16131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 04:41:51.268888 16131 server_base.cc:1034] running on GCE node
W20260925 04:41:51.268874 16136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:41:51.268960 16139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:41:51.268874 16137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:41:51.269558 16131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:41:51.270218 16131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:41:51.271378 16131 hybrid_clock.cc:648] HybridClock initialized: now 1790311311271359 us; error 37 us; skew 500 ppm
Sep 25 04:41:51 dist-test-slave-dc95 krb5kdc[15756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:41:51.274273 16131 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 04:41:51.275331 16131 webserver.cc:460] Webserver started at http://127.0.98.3:41915/ using document root <none> and password file <none>
I20260925 04:41:51.275960 16131 fs_manager.cc:359] Metadata directory not provided
I20260925 04:41:51.276036 16131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:41:51.276245 16131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:41:51.278446 16131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-2/data/instance:
uuid: "b08af7bd450046428e0a48e3edbcb12f"
format_stamp: "Formatted at 2026-09-25 04:41:51 on dist-test-slave-dc95"
server_key: "680b0018850ddae6948b89e79157a822"
server_key_iv: "204bba1af1aca76b5183c55413db3960"
server_key_version: "encryptionkey@0"
I20260925 04:41:51.279002 16131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b08af7bd450046428e0a48e3edbcb12f"
format_stamp: "Formatted at 2026-09-25 04:41:51 on dist-test-slave-dc95"
server_key: "680b0018850ddae6948b89e79157a822"
server_key_iv: "204bba1af1aca76b5183c55413db3960"
server_key_version: "encryptionkey@0"
I20260925 04:41:51.282896 16131 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260925 04:41:51.285692 16146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:41:51.287103 16131 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:41:51.287230 16131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-2/wal
uuid: "b08af7bd450046428e0a48e3edbcb12f"
format_stamp: "Formatted at 2026-09-25 04:41:51 on dist-test-slave-dc95"
server_key: "680b0018850ddae6948b89e79157a822"
server_key_iv: "204bba1af1aca76b5183c55413db3960"
server_key_version: "encryptionkey@0"
I20260925 04:41:51.287349 16131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:41:51.300874 16131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:41:51.304001 16131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:41:51.304185 16131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:41:51.304780 16131 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:41:51.305642 16131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:41:51.305692 16131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:41:51.305745 16131 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:41:51.305773 16131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:41:51.315290 16131 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40201
I20260925 04:41:51.315341 16259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40201 every 8 connection(s)
I20260925 04:41:51.316391 16131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
I20260925 04:41:51.318665 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 16131
I20260925 04:41:51.318770 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-2/wal/instance
I20260925 04:41:51.319041 392 external_mini_cluster.cc:1428] Setting key 42212a32af27f0ccbea1a3cdbb7d8208
Sep 25 04:41:51 dist-test-slave-dc95 krb5kdc[15756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:41:51.330600 15840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:51.318458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47401 (local address 127.0.98.62:38799)
0925 04:41:51.318612 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 04:41:51.318616 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:41:51.319841 (+ 1225us) server_negotiation.cc:407] Connection header received
0925 04:41:51.320916 (+ 1075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:41:51.320919 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:41:51.320975 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:41:51.321048 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:41:51.322516 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:51.323059 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:51.324337 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:51.324555 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:51.324644 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:51.327083 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:41:51.327087 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:41:51.327089 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:41:51.327116 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:41:51.328502 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:51.329035 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:51.329039 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:51.329040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:51.329121 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:51.329454 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:51.329458 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:51.329460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:51.329654 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:41:51.329743 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 04:41:51.330284 (+ 541us) server_negotiation.cc:299] Negotiation successful
0925 04:41:51.330432 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 04:41:51.331450 16262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:51.318858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38799 (local address 127.0.98.3:47401)
0925 04:41:51.319670 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:51.319729 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 04:41:51.320707 (+ 978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:41:51.321257 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:41:51.321267 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:41:51.321697 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:41:51.322360 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:51.322380 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:51.323180 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:51.323184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:51.324212 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:51.324222 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:51.324697 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:51.324702 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:51.324884 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:51.325456 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:41:51.325474 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:41:51.326896 (+ 1422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:41:51.328622 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:41:51.328630 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:41:51.328649 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:41:51.328931 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:41:51.329208 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:41:51.329212 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:41:51.329214 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:41:51.329363 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:41:51.329756 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:41:51.329764 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:41:51.330042 (+ 278us) client_negotiation.cc:769] Sending connection context
0925 04:41:51.330221 (+ 179us) client_negotiation.cc:240] Negotiation successful
0925 04:41:51.330575 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":154,"threads_started":1}
I20260925 04:41:51.332837 16260 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38799
I20260925 04:41:51.333127 16260 heartbeater.cc:464] Registering TS with master...
I20260925 04:41:51.333678 16260 heartbeater.cc:511] Master 127.0.98.62:38799 requested a full tablet report, sending...
I20260925 04:41:51.334767 15800 ts_manager.cc:194] Registered new tserver with Master: b08af7bd450046428e0a48e3edbcb12f (127.0.98.3:40201)
I20260925 04:41:51.335466 15800 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47401
I20260925 04:41:51.345479 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:41:51.355291 15840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:51.347524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36314 (local address 127.0.98.62:38799)
0925 04:41:51.347670 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 04:41:51.347676 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:41:51.347847 (+ 171us) server_negotiation.cc:407] Connection header received
0925 04:41:51.347969 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:41:51.347973 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:41:51.348020 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:41:51.348085 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:41:51.348853 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:51.349380 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:51.350236 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:51.350454 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:51.350546 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:51.352026 (+ 1480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:41:51.352029 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:41:51.352032 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:41:51.352060 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:41:51.353539 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:51.354002 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:51.354005 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:51.354006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:51.354093 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:41:51.354368 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:41:51.354372 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:41:51.354374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:41:51.354622 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:41:51.354751 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 04:41:51.355000 (+ 249us) server_negotiation.cc:299] Negotiation successful
0925 04:41:51.355128 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 04:41:51.359005 15800 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36314:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:41:51.362227 15800 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:41:51.377493 16271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:51.371741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41203 (local address 127.0.0.1:58686)
0925 04:41:51.372158 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:51.372208 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 04:41:51.372340 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:41:51.373434 (+ 1094us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:41:51.373441 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:41:51.373464 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:41:51.373607 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:51.373613 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:51.374742 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:51.374746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:51.375944 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:51.375955 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:51.376830 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:51.376834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:51.377035 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:51.377076 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 04:41:51.377176 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 04:41:51.377329 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":65,"thread_start_us":81,"threads_started":1}
I20260925 04:41:51.377501 16272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:51.371882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58686 (local address 127.0.98.1:41203)
0925 04:41:51.372675 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0925 04:41:51.372682 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:41:51.372708 (+ 26us) server_negotiation.cc:407] Connection header received
0925 04:41:51.372785 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:41:51.372791 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:41:51.373129 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:41:51.373435 (+ 306us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:41:51.373720 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:51.374631 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:51.376080 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:51.376697 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:51.376819 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:51.376993 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 04:41:51.377236 (+ 243us) server_negotiation.cc:299] Negotiation successful
0925 04:41:51.377348 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":685,"thread_start_us":86,"threads_started":1}
I20260925 04:41:51.378482 16274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:51.372465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45283 (local address 127.0.0.1:46752)
0925 04:41:51.373073 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:51.373093 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 04:41:51.373174 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:41:51.373975 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:41:51.373979 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:41:51.374001 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:41:51.374119 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:51.374125 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:51.375740 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:51.375743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:51.376886 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:51.376893 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:51.378028 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:51.378032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:51.378183 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:51.378207 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 04:41:51.378265 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 04:41:51.378331 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"mutex_wait_us":26,"thread_start_us":90,"threads_started":1}
I20260925 04:41:51.378664 16276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:51.372884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60476 (local address 127.0.98.3:40201)
0925 04:41:51.373401 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0925 04:41:51.373422 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 04:41:51.373448 (+ 26us) server_negotiation.cc:407] Connection header received
0925 04:41:51.373523 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:41:51.373528 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:41:51.373726 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:41:51.373995 (+ 269us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:41:51.374184 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:51.375358 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:51.377018 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:51.377879 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:51.377991 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:51.378162 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 04:41:51.378423 (+ 261us) server_negotiation.cc:299] Negotiation successful
0925 04:41:51.378533 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":240,"threads_started":1}
I20260925 04:41:51.378655 16273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:51.371830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40201 (local address 127.0.0.1:60476)
0925 04:41:51.372240 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:51.372254 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:41:51.372355 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:41:51.373937 (+ 1582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:41:51.373939 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:41:51.373959 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:41:51.374073 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:51.374079 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:51.375552 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:51.375556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:51.376834 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:51.376846 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:51.378024 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:51.378028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:51.378245 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:51.378283 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 04:41:51.378359 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 04:41:51.378432 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"mutex_wait_us":78,"spinlock_wait_cycles":5376,"thread_start_us":147,"threads_started":1}
I20260925 04:41:51.379254 16275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:51.372650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46752 (local address 127.0.98.2:45283)
0925 04:41:51.373427 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0925 04:41:51.373438 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 04:41:51.373456 (+ 18us) server_negotiation.cc:407] Connection header received
0925 04:41:51.373523 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:41:51.373528 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:41:51.373726 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:41:51.374276 (+ 550us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:41:51.374455 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:51.375580 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:51.377008 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:51.377879 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:51.378795 (+ 916us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:51.378960 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 04:41:51.379015 (+ 55us) server_negotiation.cc:299] Negotiation successful
0925 04:41:51.379115 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":686,"thread_start_us":108,"threads_started":1}
I20260925 04:41:51.381824 16060 tablet_service.cc:1467] Processing CreateTablet for tablet 33ec1c829bd9460a8cdcd383050518c2 (DEFAULT_TABLE table=test-table [id=00800117a2c6455cab6ed667fee91362]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:41:51.381611 15926 tablet_service.cc:1467] Processing CreateTablet for tablet 33ec1c829bd9460a8cdcd383050518c2 (DEFAULT_TABLE table=test-table [id=00800117a2c6455cab6ed667fee91362]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:41:51.382835 15926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33ec1c829bd9460a8cdcd383050518c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:41:51.382843 16060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33ec1c829bd9460a8cdcd383050518c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:41:51.382495 16193 tablet_service.cc:1467] Processing CreateTablet for tablet 33ec1c829bd9460a8cdcd383050518c2 (DEFAULT_TABLE table=test-table [id=00800117a2c6455cab6ed667fee91362]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:41:51.383780 16193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33ec1c829bd9460a8cdcd383050518c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:41:51.390928 16277 tablet_bootstrap.cc:492] T 33ec1c829bd9460a8cdcd383050518c2 P 699d9f5435bf4e2692984cf3bf4acd8c: Bootstrap starting.
I20260925 04:41:51.391932 16278 tablet_bootstrap.cc:492] T 33ec1c829bd9460a8cdcd383050518c2 P 11f96b80805a4844b6650c59f3d524eb: Bootstrap starting.
I20260925 04:41:51.393656 16277 tablet_bootstrap.cc:654] T 33ec1c829bd9460a8cdcd383050518c2 P 699d9f5435bf4e2692984cf3bf4acd8c: Neither blocks nor log segments found. Creating new log.
I20260925 04:41:51.393846 16278 tablet_bootstrap.cc:654] T 33ec1c829bd9460a8cdcd383050518c2 P 11f96b80805a4844b6650c59f3d524eb: Neither blocks nor log segments found. Creating new log.
I20260925 04:41:51.394456 16277 log.cc:824] T 33ec1c829bd9460a8cdcd383050518c2 P 699d9f5435bf4e2692984cf3bf4acd8c: Log is configured to *not* fsync() on all Append() calls
I20260925 04:41:51.394536 16278 log.cc:824] T 33ec1c829bd9460a8cdcd383050518c2 P 11f96b80805a4844b6650c59f3d524eb: Log is configured to *not* fsync() on all Append() calls
I20260925 04:41:51.395083 16279 tablet_bootstrap.cc:492] T 33ec1c829bd9460a8cdcd383050518c2 P b08af7bd450046428e0a48e3edbcb12f: Bootstrap starting.
I20260925 04:41:51.396695 16278 tablet_bootstrap.cc:492] T 33ec1c829bd9460a8cdcd383050518c2 P 11f96b80805a4844b6650c59f3d524eb: No bootstrap required, opened a new log
I20260925 04:41:51.396756 16277 tablet_bootstrap.cc:492] T 33ec1c829bd9460a8cdcd383050518c2 P 699d9f5435bf4e2692984cf3bf4acd8c: No bootstrap required, opened a new log
I20260925 04:41:51.396911 16278 ts_tablet_manager.cc:1397] T 33ec1c829bd9460a8cdcd383050518c2 P 11f96b80805a4844b6650c59f3d524eb: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 04:41:51.396935 16277 ts_tablet_manager.cc:1397] T 33ec1c829bd9460a8cdcd383050518c2 P 699d9f5435bf4e2692984cf3bf4acd8c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 04:41:51.397743 16279 tablet_bootstrap.cc:654] T 33ec1c829bd9460a8cdcd383050518c2 P b08af7bd450046428e0a48e3edbcb12f: Neither blocks nor log segments found. Creating new log.
I20260925 04:41:51.399281 16279 log.cc:824] T 33ec1c829bd9460a8cdcd383050518c2 P b08af7bd450046428e0a48e3edbcb12f: Log is configured to *not* fsync() on all Append() calls
I20260925 04:41:51.401048 16278 raft_consensus.cc:348] T 33ec1c829bd9460a8cdcd383050518c2 P 11f96b80805a4844b6650c59f3d524eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f96b80805a4844b6650c59f3d524eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41203 } } peers { permanent_uuid: "b08af7bd450046428e0a48e3edbcb12f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40201 } } peers { permanent_uuid: "699d9f5435bf4e2692984cf3bf4acd8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45283 } }
I20260925 04:41:51.400949 16277 raft_consensus.cc:348] T 33ec1c829bd9460a8cdcd383050518c2 P 699d9f5435bf4e2692984cf3bf4acd8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f96b80805a4844b6650c59f3d524eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41203 } } peers { permanent_uuid: "b08af7bd450046428e0a48e3edbcb12f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40201 } } peers { permanent_uuid: "699d9f5435bf4e2692984cf3bf4acd8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45283 } }
I20260925 04:41:51.401286 16278 raft_consensus.cc:374] T 33ec1c829bd9460a8cdcd383050518c2 P 11f96b80805a4844b6650c59f3d524eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:41:51.401289 16277 raft_consensus.cc:374] T 33ec1c829bd9460a8cdcd383050518c2 P 699d9f5435bf4e2692984cf3bf4acd8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:41:51.401357 16277 raft_consensus.cc:729] T 33ec1c829bd9460a8cdcd383050518c2 P 699d9f5435bf4e2692984cf3bf4acd8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 699d9f5435bf4e2692984cf3bf4acd8c, State: Initialized, Role: FOLLOWER
I20260925 04:41:51.401374 16278 raft_consensus.cc:729] T 33ec1c829bd9460a8cdcd383050518c2 P 11f96b80805a4844b6650c59f3d524eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11f96b80805a4844b6650c59f3d524eb, State: Initialized, Role: FOLLOWER
I20260925 04:41:51.401902 16279 tablet_bootstrap.cc:492] T 33ec1c829bd9460a8cdcd383050518c2 P b08af7bd450046428e0a48e3edbcb12f: No bootstrap required, opened a new log
I20260925 04:41:51.401865 16278 consensus_queue.cc:260] T 33ec1c829bd9460a8cdcd383050518c2 P 11f96b80805a4844b6650c59f3d524eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f96b80805a4844b6650c59f3d524eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41203 } } peers { permanent_uuid: "b08af7bd450046428e0a48e3edbcb12f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40201 } } peers { permanent_uuid: "699d9f5435bf4e2692984cf3bf4acd8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45283 } }
I20260925 04:41:51.401885 16277 consensus_queue.cc:260] T 33ec1c829bd9460a8cdcd383050518c2 P 699d9f5435bf4e2692984cf3bf4acd8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f96b80805a4844b6650c59f3d524eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41203 } } peers { permanent_uuid: "b08af7bd450046428e0a48e3edbcb12f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40201 } } peers { permanent_uuid: "699d9f5435bf4e2692984cf3bf4acd8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45283 } }
I20260925 04:41:51.402159 16279 ts_tablet_manager.cc:1397] T 33ec1c829bd9460a8cdcd383050518c2 P b08af7bd450046428e0a48e3edbcb12f: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20260925 04:41:51.403004 16277 ts_tablet_manager.cc:1428] T 33ec1c829bd9460a8cdcd383050518c2 P 699d9f5435bf4e2692984cf3bf4acd8c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 04:41:51.403007 16278 ts_tablet_manager.cc:1428] T 33ec1c829bd9460a8cdcd383050518c2 P 11f96b80805a4844b6650c59f3d524eb: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 04:41:51.403416 15992 heartbeater.cc:503] Master 127.0.98.62:38799 was elected leader, sending a full tablet report...
I20260925 04:41:51.403415 16126 heartbeater.cc:503] Master 127.0.98.62:38799 was elected leader, sending a full tablet report...
I20260925 04:41:51.405978 16279 raft_consensus.cc:348] T 33ec1c829bd9460a8cdcd383050518c2 P b08af7bd450046428e0a48e3edbcb12f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f96b80805a4844b6650c59f3d524eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41203 } } peers { permanent_uuid: "b08af7bd450046428e0a48e3edbcb12f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40201 } } peers { permanent_uuid: "699d9f5435bf4e2692984cf3bf4acd8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45283 } }
I20260925 04:41:51.406248 16279 raft_consensus.cc:374] T 33ec1c829bd9460a8cdcd383050518c2 P b08af7bd450046428e0a48e3edbcb12f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:41:51.406334 16279 raft_consensus.cc:729] T 33ec1c829bd9460a8cdcd383050518c2 P b08af7bd450046428e0a48e3edbcb12f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b08af7bd450046428e0a48e3edbcb12f, State: Initialized, Role: FOLLOWER
I20260925 04:41:51.406845 16279 consensus_queue.cc:260] T 33ec1c829bd9460a8cdcd383050518c2 P b08af7bd450046428e0a48e3edbcb12f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f96b80805a4844b6650c59f3d524eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41203 } } peers { permanent_uuid: "b08af7bd450046428e0a48e3edbcb12f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40201 } } peers { permanent_uuid: "699d9f5435bf4e2692984cf3bf4acd8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45283 } }
I20260925 04:41:51.407596 16260 heartbeater.cc:503] Master 127.0.98.62:38799 was elected leader, sending a full tablet report...
I20260925 04:41:51.408716 16279 ts_tablet_manager.cc:1428] T 33ec1c829bd9460a8cdcd383050518c2 P b08af7bd450046428e0a48e3edbcb12f: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260925 04:41:51.502053 16284 raft_consensus.cc:482] T 33ec1c829bd9460a8cdcd383050518c2 P 11f96b80805a4844b6650c59f3d524eb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:41:51.502259 16284 raft_consensus.cc:504] T 33ec1c829bd9460a8cdcd383050518c2 P 11f96b80805a4844b6650c59f3d524eb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f96b80805a4844b6650c59f3d524eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41203 } } peers { permanent_uuid: "b08af7bd450046428e0a48e3edbcb12f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40201 } } peers { permanent_uuid: "699d9f5435bf4e2692984cf3bf4acd8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45283 } }
I20260925 04:41:51.503402 16284 leader_election.cc:288] T 33ec1c829bd9460a8cdcd383050518c2 P 11f96b80805a4844b6650c59f3d524eb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b08af7bd450046428e0a48e3edbcb12f (127.0.98.3:40201), 699d9f5435bf4e2692984cf3bf4acd8c (127.0.98.2:45283)
I20260925 04:41:51.507037 16285 raft_consensus.cc:482] T 33ec1c829bd9460a8cdcd383050518c2 P b08af7bd450046428e0a48e3edbcb12f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:41:51.507247 16285 raft_consensus.cc:504] T 33ec1c829bd9460a8cdcd383050518c2 P b08af7bd450046428e0a48e3edbcb12f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f96b80805a4844b6650c59f3d524eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41203 } } peers { permanent_uuid: "b08af7bd450046428e0a48e3edbcb12f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40201 } } peers { permanent_uuid: "699d9f5435bf4e2692984cf3bf4acd8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45283 } }
I20260925 04:41:51.508006 16276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:51.503766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37427 (local address 127.0.98.3:40201)
0925 04:41:51.503895 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 04:41:51.503899 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:41:51.504094 (+ 195us) server_negotiation.cc:407] Connection header received
0925 04:41:51.504275 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:41:51.504279 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:41:51.504334 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:41:51.504381 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:41:51.504744 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:51.505490 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:51.506773 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:51.507287 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:51.507406 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:51.507473 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 04:41:51.507805 (+ 332us) server_negotiation.cc:299] Negotiation successful
0925 04:41:51.507894 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 04:41:51.508035 16286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:51.503694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40201 (local address 127.0.98.1:37427)
0925 04:41:51.503996 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:51.504010 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:41:51.504127 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:41:51.504506 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:41:51.504508 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:41:51.504537 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:41:51.504657 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:51.504661 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:51.505636 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:51.505640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:51.506611 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:51.506623 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:51.507538 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:51.507542 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:51.507651 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:51.507678 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 04:41:51.507781 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 04:41:51.507863 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":44,"spinlock_wait_cycles":32128,"thread_start_us":109,"threads_started":1}
I20260925 04:41:51.508118 16275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:51.504279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50433 (local address 127.0.98.2:45283)
0925 04:41:51.504407 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 04:41:51.504411 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:41:51.504513 (+ 102us) server_negotiation.cc:407] Connection header received
0925 04:41:51.504712 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:41:51.504715 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:41:51.504766 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:41:51.504842 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:41:51.505088 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:51.505729 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:51.506800 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:51.507345 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:51.507444 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:51.507508 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 04:41:51.507852 (+ 344us) server_negotiation.cc:299] Negotiation successful
0925 04:41:51.507930 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 04:41:51.508531 16287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:51.504113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45283 (local address 127.0.98.1:50433)
0925 04:41:51.504415 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:51.504428 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:41:51.504600 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:41:51.504883 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:41:51.504885 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:41:51.504898 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:41:51.504980 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:51.504984 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:51.505837 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:51.505840 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:51.506671 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:51.506681 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:51.507572 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:51.507575 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:51.507710 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:51.507732 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 04:41:51.508309 (+ 577us) client_negotiation.cc:240] Negotiation successful
0925 04:41:51.508389 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":22,"thread_start_us":95,"threads_started":1}
I20260925 04:41:51.508580 16214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33ec1c829bd9460a8cdcd383050518c2" candidate_uuid: "11f96b80805a4844b6650c59f3d524eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b08af7bd450046428e0a48e3edbcb12f" is_pre_election: true
I20260925 04:41:51.508697 16285 leader_election.cc:288] T 33ec1c829bd9460a8cdcd383050518c2 P b08af7bd450046428e0a48e3edbcb12f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 11f96b80805a4844b6650c59f3d524eb (127.0.98.1:41203), 699d9f5435bf4e2692984cf3bf4acd8c (127.0.98.2:45283)
I20260925 04:41:51.508828 16214 raft_consensus.cc:2454] T 33ec1c829bd9460a8cdcd383050518c2 P b08af7bd450046428e0a48e3edbcb12f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11f96b80805a4844b6650c59f3d524eb in term 0.
I20260925 04:41:51.509270 16080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33ec1c829bd9460a8cdcd383050518c2" candidate_uuid: "11f96b80805a4844b6650c59f3d524eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "699d9f5435bf4e2692984cf3bf4acd8c" is_pre_election: true
I20260925 04:41:51.509407 15882 leader_election.cc:302] T 33ec1c829bd9460a8cdcd383050518c2 P 11f96b80805a4844b6650c59f3d524eb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11f96b80805a4844b6650c59f3d524eb, b08af7bd450046428e0a48e3edbcb12f; no voters:
I20260925 04:41:51.509585 16080 raft_consensus.cc:2454] T 33ec1c829bd9460a8cdcd383050518c2 P 699d9f5435bf4e2692984cf3bf4acd8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11f96b80805a4844b6650c59f3d524eb in term 0.
I20260925 04:41:51.509724 16284 raft_consensus.cc:2781] T 33ec1c829bd9460a8cdcd383050518c2 P 11f96b80805a4844b6650c59f3d524eb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:41:51.509809 16284 raft_consensus.cc:482] T 33ec1c829bd9460a8cdcd383050518c2 P 11f96b80805a4844b6650c59f3d524eb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:41:51.509871 16284 raft_consensus.cc:3037] T 33ec1c829bd9460a8cdcd383050518c2 P 11f96b80805a4844b6650c59f3d524eb [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:41:51.511288 16284 raft_consensus.cc:504] T 33ec1c829bd9460a8cdcd383050518c2 P 11f96b80805a4844b6650c59f3d524eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f96b80805a4844b6650c59f3d524eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41203 } } peers { permanent_uuid: "b08af7bd450046428e0a48e3edbcb12f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40201 } } peers { permanent_uuid: "699d9f5435bf4e2692984cf3bf4acd8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45283 } }
I20260925 04:41:51.511739 16284 leader_election.cc:288] T 33ec1c829bd9460a8cdcd383050518c2 P 11f96b80805a4844b6650c59f3d524eb [CANDIDATE]: Term 1 election: Requested vote from peers b08af7bd450046428e0a48e3edbcb12f (127.0.98.3:40201), 699d9f5435bf4e2692984cf3bf4acd8c (127.0.98.2:45283)
I20260925 04:41:51.512336 16080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33ec1c829bd9460a8cdcd383050518c2" candidate_uuid: "11f96b80805a4844b6650c59f3d524eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "699d9f5435bf4e2692984cf3bf4acd8c"
I20260925 04:41:51.512436 16214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33ec1c829bd9460a8cdcd383050518c2" candidate_uuid: "11f96b80805a4844b6650c59f3d524eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b08af7bd450046428e0a48e3edbcb12f"
I20260925 04:41:51.512481 16080 raft_consensus.cc:3037] T 33ec1c829bd9460a8cdcd383050518c2 P 699d9f5435bf4e2692984cf3bf4acd8c [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:41:51.512558 16214 raft_consensus.cc:3037] T 33ec1c829bd9460a8cdcd383050518c2 P b08af7bd450046428e0a48e3edbcb12f [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:41:51.513405 16262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:51.508987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41203 (local address 127.0.98.3:52269)
0925 04:41:51.509226 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:51.509240 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:41:51.509318 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:41:51.509832 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:41:51.509835 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:41:51.509872 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:41:51.509987 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:51.509994 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:51.510899 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:51.510903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:51.511873 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:51.511884 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:51.512971 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:51.512975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:51.513084 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:51.513112 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 04:41:51.513180 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 04:41:51.513259 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169}
I20260925 04:41:51.513846 16214 raft_consensus.cc:2454] T 33ec1c829bd9460a8cdcd383050518c2 P b08af7bd450046428e0a48e3edbcb12f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11f96b80805a4844b6650c59f3d524eb in term 1.
I20260925 04:41:51.513893 16272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:51.509208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52269 (local address 127.0.98.1:41203)
0925 04:41:51.509552 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0925 04:41:51.509557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:41:51.509570 (+ 13us) server_negotiation.cc:407] Connection header received
0925 04:41:51.509611 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:41:51.509614 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:41:51.509663 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:41:51.509732 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:41:51.510111 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:51.510790 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:51.512157 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:51.512691 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:51.512841 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:51.512909 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 04:41:51.513670 (+ 761us) server_negotiation.cc:299] Negotiation successful
0925 04:41:51.513751 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":271}
I20260925 04:41:51.513862 16080 raft_consensus.cc:2454] T 33ec1c829bd9460a8cdcd383050518c2 P 699d9f5435bf4e2692984cf3bf4acd8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11f96b80805a4844b6650c59f3d524eb in term 1.
I20260925 04:41:51.514091 16288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:51.509195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45283 (local address 127.0.98.3:49789)
0925 04:41:51.509673 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:51.509686 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:41:51.509775 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:41:51.510345 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:41:51.510348 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:41:51.510387 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:41:51.510491 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:51.510498 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:51.511461 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:51.511464 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:51.512560 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:41:51.512569 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:41:51.513524 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:41:51.513528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:41:51.513621 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:51.513642 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 04:41:51.513889 (+ 247us) client_negotiation.cc:240] Negotiation successful
0925 04:41:51.513961 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":122,"threads_started":1}
I20260925 04:41:51.514250 15882 leader_election.cc:302] T 33ec1c829bd9460a8cdcd383050518c2 P 11f96b80805a4844b6650c59f3d524eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11f96b80805a4844b6650c59f3d524eb, b08af7bd450046428e0a48e3edbcb12f; no voters:
I20260925 04:41:51.514413 16284 raft_consensus.cc:2781] T 33ec1c829bd9460a8cdcd383050518c2 P 11f96b80805a4844b6650c59f3d524eb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:41:51.514539 16275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:51.509896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49789 (local address 127.0.98.2:45283)
0925 04:41:51.510074 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 04:41:51.510078 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:41:51.510092 (+ 14us) server_negotiation.cc:407] Connection header received
0925 04:41:51.510133 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:41:51.510136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:41:51.510178 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:41:51.510241 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:41:51.510613 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:51.511298 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:51.512698 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:51.513245 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:51.513358 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:51.513415 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 04:41:51.513762 (+ 347us) server_negotiation.cc:299] Negotiation successful
0925 04:41:51.513840 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260925 04:41:51.514664 16284 raft_consensus.cc:686] T 33ec1c829bd9460a8cdcd383050518c2 P 11f96b80805a4844b6650c59f3d524eb [term 1 LEADER]: Becoming Leader. State: Replica: 11f96b80805a4844b6650c59f3d524eb, State: Running, Role: LEADER
I20260925 04:41:51.514870 15946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33ec1c829bd9460a8cdcd383050518c2" candidate_uuid: "b08af7bd450046428e0a48e3edbcb12f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11f96b80805a4844b6650c59f3d524eb" is_pre_election: true
I20260925 04:41:51.514922 16080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33ec1c829bd9460a8cdcd383050518c2" candidate_uuid: "b08af7bd450046428e0a48e3edbcb12f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "699d9f5435bf4e2692984cf3bf4acd8c" is_pre_election: true
I20260925 04:41:51.515091 16080 raft_consensus.cc:2379] T 33ec1c829bd9460a8cdcd383050518c2 P 699d9f5435bf4e2692984cf3bf4acd8c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b08af7bd450046428e0a48e3edbcb12f in current term 1: Already voted for candidate 11f96b80805a4844b6650c59f3d524eb in this term.
I20260925 04:41:51.515039 16284 consensus_queue.cc:237] T 33ec1c829bd9460a8cdcd383050518c2 P 11f96b80805a4844b6650c59f3d524eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f96b80805a4844b6650c59f3d524eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41203 } } peers { permanent_uuid: "b08af7bd450046428e0a48e3edbcb12f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40201 } } peers { permanent_uuid: "699d9f5435bf4e2692984cf3bf4acd8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45283 } }
I20260925 04:41:51.515677 16149 leader_election.cc:302] T 33ec1c829bd9460a8cdcd383050518c2 P b08af7bd450046428e0a48e3edbcb12f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b08af7bd450046428e0a48e3edbcb12f; no voters: 11f96b80805a4844b6650c59f3d524eb, 699d9f5435bf4e2692984cf3bf4acd8c
I20260925 04:41:51.515952 16285 raft_consensus.cc:2726] T 33ec1c829bd9460a8cdcd383050518c2 P b08af7bd450046428e0a48e3edbcb12f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 04:41:51.518539 15800 catalog_manager.cc:5495] T 33ec1c829bd9460a8cdcd383050518c2 P 11f96b80805a4844b6650c59f3d524eb reported cstate change: term changed from 0 to 1, leader changed from <none> to 11f96b80805a4844b6650c59f3d524eb (127.0.98.1). New cstate: current_term: 1 leader_uuid: "11f96b80805a4844b6650c59f3d524eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f96b80805a4844b6650c59f3d524eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b08af7bd450046428e0a48e3edbcb12f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "699d9f5435bf4e2692984cf3bf4acd8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45283 } health_report { overall_health: UNKNOWN } } }
I20260925 04:41:51.533495 16272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:51.530023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58702 (local address 127.0.98.1:41203)
0925 04:41:51.530160 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 04:41:51.530164 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:41:51.530179 (+ 15us) server_negotiation.cc:407] Connection header received
0925 04:41:51.530262 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:41:51.530265 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:41:51.530320 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:41:51.530409 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:41:51.530655 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:51.531186 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:51.532046 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:41:51.532257 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:41:51.532342 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:41:51.532644 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:41:51.533085 (+ 441us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:41:51.533191 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 04:41:51.533276 (+ 85us) server_negotiation.cc:299] Negotiation successful
0925 04:41:51.533351 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 04:41:51.539824 16080 raft_consensus.cc:1261] T 33ec1c829bd9460a8cdcd383050518c2 P 699d9f5435bf4e2692984cf3bf4acd8c [term 1 FOLLOWER]: Refusing update from remote peer 11f96b80805a4844b6650c59f3d524eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:41:51.539933 16214 raft_consensus.cc:1261] T 33ec1c829bd9460a8cdcd383050518c2 P b08af7bd450046428e0a48e3edbcb12f [term 1 FOLLOWER]: Refusing update from remote peer 11f96b80805a4844b6650c59f3d524eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:41:51.540655 16284 consensus_queue.cc:1035] T 33ec1c829bd9460a8cdcd383050518c2 P 11f96b80805a4844b6650c59f3d524eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b08af7bd450046428e0a48e3edbcb12f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:41:51.540879 16289 consensus_queue.cc:1035] T 33ec1c829bd9460a8cdcd383050518c2 P 11f96b80805a4844b6650c59f3d524eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "699d9f5435bf4e2692984cf3bf4acd8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:41:51.549003 16293 mvcc.cc:205] Tried to move back new op lower bound from 7333115132059185152 to 7333115131968909312. Current Snapshot: MvccSnapshot[applied={T|T < 7333115132059185152}]
I20260925 04:41:51.550096 16294 mvcc.cc:205] Tried to move back new op lower bound from 7333115132059185152 to 7333115131968909312. Current Snapshot: MvccSnapshot[applied={T|T < 7333115132059185152}]
I20260925 04:41:51.557444 15800 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:41:51.557621 15800 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:41:51.560315 15800 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7fc2639fe7ca454c8de7a2cfce0e63c3: Sending DeleteTablet for 3 replicas of tablet 33ec1c829bd9460a8cdcd383050518c2
I20260925 04:41:51.561188 15926 tablet_service.cc:1514] Processing DeleteTablet for tablet 33ec1c829bd9460a8cdcd383050518c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:41:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58686
I20260925 04:41:51.561203 16193 tablet_service.cc:1514] Processing DeleteTablet for tablet 33ec1c829bd9460a8cdcd383050518c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:41:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60476
I20260925 04:41:51.561928 16301 tablet_replica.cc:330] T 33ec1c829bd9460a8cdcd383050518c2 P b08af7bd450046428e0a48e3edbcb12f: stopping tablet replica
I20260925 04:41:51.562186 16302 tablet_replica.cc:330] T 33ec1c829bd9460a8cdcd383050518c2 P 11f96b80805a4844b6650c59f3d524eb: stopping tablet replica
I20260925 04:41:51.562153 16060 tablet_service.cc:1514] Processing DeleteTablet for tablet 33ec1c829bd9460a8cdcd383050518c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:41:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46752
I20260925 04:41:51.562583 16302 raft_consensus.cc:2229] T 33ec1c829bd9460a8cdcd383050518c2 P 11f96b80805a4844b6650c59f3d524eb [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:41:51.562991 16302 raft_consensus.cc:2258] T 33ec1c829bd9460a8cdcd383050518c2 P 11f96b80805a4844b6650c59f3d524eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:41:51.563122 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 15863
I20260925 04:41:51.563464 16301 raft_consensus.cc:2229] T 33ec1c829bd9460a8cdcd383050518c2 P b08af7bd450046428e0a48e3edbcb12f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:41:51.563819 16301 raft_consensus.cc:2258] T 33ec1c829bd9460a8cdcd383050518c2 P b08af7bd450046428e0a48e3edbcb12f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:41:51.565400 16301 ts_tablet_manager.cc:1905] T 33ec1c829bd9460a8cdcd383050518c2 P b08af7bd450046428e0a48e3edbcb12f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:41:51.568782 16301 ts_tablet_manager.cc:1918] T 33ec1c829bd9460a8cdcd383050518c2 P b08af7bd450046428e0a48e3edbcb12f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:41:51.568894 16301 log.cc:1196] T 33ec1c829bd9460a8cdcd383050518c2 P b08af7bd450046428e0a48e3edbcb12f: Deleting WAL directory at /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-2/wal/wals/33ec1c829bd9460a8cdcd383050518c2
I20260925 04:41:51.569267 16301 ts_tablet_manager.cc:1939] T 33ec1c829bd9460a8cdcd383050518c2 P b08af7bd450046428e0a48e3edbcb12f: Deleting consensus metadata
I20260925 04:41:51.569669 16303 tablet_replica.cc:330] T 33ec1c829bd9460a8cdcd383050518c2 P 699d9f5435bf4e2692984cf3bf4acd8c: stopping tablet replica
I20260925 04:41:51.570087 16303 raft_consensus.cc:2229] T 33ec1c829bd9460a8cdcd383050518c2 P 699d9f5435bf4e2692984cf3bf4acd8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:41:51.570513 16303 raft_consensus.cc:2258] T 33ec1c829bd9460a8cdcd383050518c2 P 699d9f5435bf4e2692984cf3bf4acd8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:41:51.571113 15788 catalog_manager.cc:4841] TS b08af7bd450046428e0a48e3edbcb12f (127.0.98.3:40201): tablet 33ec1c829bd9460a8cdcd383050518c2 (table test-table [id=00800117a2c6455cab6ed667fee91362]) successfully deleted
W20260925 04:41:51.571441 15787 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41203 (error 108)
I20260925 04:41:51.572301 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 15997
I20260925 04:41:51.572400 16303 ts_tablet_manager.cc:1905] T 33ec1c829bd9460a8cdcd383050518c2 P 699d9f5435bf4e2692984cf3bf4acd8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:41:51.573714 16273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:51.573381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41203 (local address 127.0.0.1:58710)
0925 04:41:51.573566 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:51.573629 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
W20260925 04:41:51.573872 15787 catalog_manager.cc:4584] TS 11f96b80805a4844b6650c59f3d524eb (127.0.98.1:41203): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 33ec1c829bd9460a8cdcd383050518c2: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41203: connect: Connection refused (error 111)
I20260925 04:41:51.577911 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 16131
I20260925 04:41:51.579650 16273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:41:51.578159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45283 (local address 127.0.0.1:46756)
0925 04:41:51.579478 (+ 1319us) negotiation.cc:107] Waiting for socket to connect
0925 04:41:51.579535 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:45283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1227}
I20260925 04:41:51.583271 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 15769
I20260925 04:42:11.588973 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38799
--webserver_interface=127.0.98.62
--webserver_port=43981
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38799
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:42:11.699753 16305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:42:11.700050 16305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:42:11.700110 16305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:42:11.704316 16305 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 04:42:11.704388 16305 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 04:42:11.704411 16305 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 04:42:11.704428 16305 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:42:11.704455 16305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:42:11.704488 16305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:42:11.704520 16305 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:42:11.704548 16305 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:42:11.710202 16305 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=43981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:42:11.711371 16305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:42:11.712435 16305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:42:11.726680 16312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:42:11.726691 16311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:42:11.726943 16305 server_base.cc:1034] running on GCE node
W20260925 04:42:11.727191 16314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:42:11.728184 16305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:42:11.729893 16305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:42:11.731097 16305 hybrid_clock.cc:648] HybridClock initialized: now 1790311331731065 us; error 52 us; skew 500 ppm
Sep 25 04:42:11 dist-test-slave-dc95 krb5kdc[15756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311331, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:42:11.734884 16305 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 04:42:11.736254 16305 webserver.cc:460] Webserver started at http://127.0.98.62:43981/ using document root <none> and password file <none>
I20260925 04:42:11.736929 16305 fs_manager.cc:359] Metadata directory not provided
I20260925 04:42:11.737035 16305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:42:11.742322 16305 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 04:42:11.745314 16321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:11.747049 16305 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 04:42:11.747336 16305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/master-0/wal
uuid: "7fc2639fe7ca454c8de7a2cfce0e63c3"
format_stamp: "Formatted at 2026-09-25 04:41:50 on dist-test-slave-dc95"
server_key: "7541e435ee78e02d5809b5333f7ccd0c"
server_key_iv: "8dfe44d99c78a72e7614d359866b216c"
server_key_version: "encryptionkey@0"
I20260925 04:42:11.748232 16305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:42:11.765396 16305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:42:11.769657 16305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:42:11.769884 16305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:42:11.777822 16305 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38799
I20260925 04:42:11.778075 16373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38799 every 8 connection(s)
I20260925 04:42:11.778961 16305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
I20260925 04:42:11.783545 16374 sys_catalog.cc:263] Verifying existing consensus state
I20260925 04:42:11.785446 16374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fc2639fe7ca454c8de7a2cfce0e63c3: Bootstrap starting.
I20260925 04:42:11.785792 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 16305
I20260925 04:42:11.786280 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41203
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:38799
--webserver_port=35771
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:42:11.808743 16374 log.cc:824] T 00000000000000000000000000000000 P 7fc2639fe7ca454c8de7a2cfce0e63c3: Log is configured to *not* fsync() on all Append() calls
I20260925 04:42:11.820658 16374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fc2639fe7ca454c8de7a2cfce0e63c3: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 04:42:11.821141 16374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fc2639fe7ca454c8de7a2cfce0e63c3: Bootstrap complete.
I20260925 04:42:11.826553 16374 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7fc2639fe7ca454c8de7a2cfce0e63c3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fc2639fe7ca454c8de7a2cfce0e63c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38799 } }
I20260925 04:42:11.827101 16374 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7fc2639fe7ca454c8de7a2cfce0e63c3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fc2639fe7ca454c8de7a2cfce0e63c3, State: Initialized, Role: FOLLOWER
I20260925 04:42:11.827713 16374 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7fc2639fe7ca454c8de7a2cfce0e63c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fc2639fe7ca454c8de7a2cfce0e63c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38799 } }
I20260925 04:42:11.827865 16374 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7fc2639fe7ca454c8de7a2cfce0e63c3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:42:11.827940 16374 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7fc2639fe7ca454c8de7a2cfce0e63c3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:42:11.828037 16374 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7fc2639fe7ca454c8de7a2cfce0e63c3 [term 1 FOLLOWER]: Advancing to term 2
I20260925 04:42:11.830734 16374 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7fc2639fe7ca454c8de7a2cfce0e63c3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fc2639fe7ca454c8de7a2cfce0e63c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38799 } }
I20260925 04:42:11.831152 16374 leader_election.cc:302] T 00000000000000000000000000000000 P 7fc2639fe7ca454c8de7a2cfce0e63c3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7fc2639fe7ca454c8de7a2cfce0e63c3; no voters:
I20260925 04:42:11.831525 16374 leader_election.cc:288] T 00000000000000000000000000000000 P 7fc2639fe7ca454c8de7a2cfce0e63c3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 04:42:11.831775 16378 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7fc2639fe7ca454c8de7a2cfce0e63c3 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 04:42:11.832041 16378 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7fc2639fe7ca454c8de7a2cfce0e63c3 [term 2 LEADER]: Becoming Leader. State: Replica: 7fc2639fe7ca454c8de7a2cfce0e63c3, State: Running, Role: LEADER
I20260925 04:42:11.832418 16378 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7fc2639fe7ca454c8de7a2cfce0e63c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fc2639fe7ca454c8de7a2cfce0e63c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38799 } }
I20260925 04:42:11.832934 16374 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7fc2639fe7ca454c8de7a2cfce0e63c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:42:11.833823 16379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fc2639fe7ca454c8de7a2cfce0e63c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7fc2639fe7ca454c8de7a2cfce0e63c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fc2639fe7ca454c8de7a2cfce0e63c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38799 } } }
I20260925 04:42:11.833947 16379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fc2639fe7ca454c8de7a2cfce0e63c3 [sys.catalog]: This master's current role is: LEADER
I20260925 04:42:11.834710 16380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fc2639fe7ca454c8de7a2cfce0e63c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7fc2639fe7ca454c8de7a2cfce0e63c3. Latest consensus state: current_term: 2 leader_uuid: "7fc2639fe7ca454c8de7a2cfce0e63c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fc2639fe7ca454c8de7a2cfce0e63c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38799 } } }
I20260925 04:42:11.834836 16380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fc2639fe7ca454c8de7a2cfce0e63c3 [sys.catalog]: This master's current role is: LEADER
I20260925 04:42:11.835090 16387 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:42:11.840555 16387 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:42:11.841348 16387 catalog_manager.cc:1259] Loaded cluster ID: 04ecd207d1c04912833509ed3e6cccae
I20260925 04:42:11.841423 16387 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:42:11.844156 16387 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:42:11.845023 16387 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7fc2639fe7ca454c8de7a2cfce0e63c3: Loaded TSK: 0
I20260925 04:42:11.860154 16387 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7fc2639fe7ca454c8de7a2cfce0e63c3: Generated new TSK 1
I20260925 04:42:11.860715 16387 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 04:42:11.919368 16376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:42:11.919718 16376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:42:11.919780 16376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:42:11.924618 16376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:42:11.924700 16376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:42:11.924795 16376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 04:42:11.930423 16376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41203
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35771
--tserver_master_addrs=127.0.98.62:38799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:42:11.931509 16376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:42:11.932595 16376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:42:11.943272 16404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:42:11.943265 16401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:42:11.943324 16402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:42:11.943722 16376 server_base.cc:1034] running on GCE node
I20260925 04:42:11.944200 16376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:42:11.945013 16376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:42:11.946240 16376 hybrid_clock.cc:648] HybridClock initialized: now 1790311331946212 us; error 42 us; skew 500 ppm
Sep 25 04:42:11 dist-test-slave-dc95 krb5kdc[15756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311331, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:42:11.949180 16376 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 04:42:11.950120 16376 webserver.cc:460] Webserver started at http://127.0.98.1:35771/ using document root <none> and password file <none>
I20260925 04:42:11.950663 16376 fs_manager.cc:359] Metadata directory not provided
I20260925 04:42:11.950742 16376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:42:11.955008 16376 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:42:11.957492 16411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:11.958899 16376 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 04:42:11.959048 16376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-0/wal
uuid: "11f96b80805a4844b6650c59f3d524eb"
format_stamp: "Formatted at 2026-09-25 04:41:50 on dist-test-slave-dc95"
server_key: "13b385f8ae8c81800a5364e5d1a66da3"
server_key_iv: "c52952f308cdd5d540d4915cc7bfa80b"
server_key_version: "encryptionkey@0"
I20260925 04:42:11.959662 16376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:42:11.970278 16376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:42:11.974138 16376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:42:11.974335 16376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:42:11.974972 16376 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:42:11.976245 16418 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 04:42:11.981977 16376 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 04:42:11.982103 16376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260925 04:42:11.982185 16376 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 04:42:11.984127 16376 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 04:42:11.984190 16376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.002s
I20260925 04:42:11.984620 16418 tablet_bootstrap.cc:492] T 33ec1c829bd9460a8cdcd383050518c2 P 11f96b80805a4844b6650c59f3d524eb: Bootstrap starting.
I20260925 04:42:11.994504 16376 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41203
I20260925 04:42:11.994596 16525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41203 every 8 connection(s)
I20260925 04:42:11.995528 16376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
I20260925 04:42:12.004179 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 16376
I20260925 04:42:12.004583 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:45283
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:38799
--webserver_port=46187
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:42:12.005116 16418 log.cc:824] T 33ec1c829bd9460a8cdcd383050518c2 P 11f96b80805a4844b6650c59f3d524eb: Log is configured to *not* fsync() on all Append() calls
Sep 25 04:42:12 dist-test-slave-dc95 krb5kdc[15756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311331, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:42:12.013624 16418 tablet_bootstrap.cc:492] T 33ec1c829bd9460a8cdcd383050518c2 P 11f96b80805a4844b6650c59f3d524eb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 04:42:12.013988 16418 tablet_bootstrap.cc:492] T 33ec1c829bd9460a8cdcd383050518c2 P 11f96b80805a4844b6650c59f3d524eb: Bootstrap complete.
I20260925 04:42:12.015043 16418 ts_tablet_manager.cc:1397] T 33ec1c829bd9460a8cdcd383050518c2 P 11f96b80805a4844b6650c59f3d524eb: Time spent bootstrapping tablet: real 0.031s user 0.018s sys 0.009s
I20260925 04:42:12.015445 16529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:11.997610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38799 (local address 127.0.98.1:54167)
0925 04:42:11.998243 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:11.998310 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0925 04:42:11.999424 (+ 1114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:42:12.000642 (+ 1218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:42:12.000651 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:42:12.001204 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:42:12.002045 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:12.002070 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:12.003541 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:12.003545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:12.004880 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:12.004893 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:12.005966 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:12.005970 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:12.006166 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:12.006859 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:42:12.006879 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:42:12.008694 (+ 1815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:42:12.011432 (+ 2738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:42:12.011439 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:42:12.011454 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:42:12.011850 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:42:12.012299 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:42:12.012302 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:42:12.012304 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:42:12.012499 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:42:12.013246 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:42:12.013252 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:42:12.013615 (+ 363us) client_negotiation.cc:769] Sending connection context
0925 04:42:12.013823 (+ 208us) client_negotiation.cc:240] Negotiation successful
0925 04:42:12.014351 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":116,"threads_started":1}
I20260925 04:42:12.017268 16528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:11.997622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54167 (local address 127.0.98.62:38799)
0925 04:42:11.998023 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:11.998043 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:11.998456 (+ 413us) server_negotiation.cc:407] Connection header received
0925 04:42:11.999896 (+ 1440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:11.999910 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:12.000218 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:12.000512 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:42:12.002286 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:12.003317 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:12.005062 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:12.005801 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:12.005998 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:12.008927 (+ 2929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:42:12.008934 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:42:12.008951 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:42:12.009000 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:42:12.011272 (+ 2272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:42:12.012011 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:42:12.012019 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:42:12.012024 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:42:12.012197 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:42:12.012615 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:42:12.012618 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:42:12.012620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:42:12.013119 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:42:12.013254 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:12.016018 (+ 2764us) server_negotiation.cc:299] Negotiation successful
0925 04:42:12.016364 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":124,"threads_started":1}
I20260925 04:42:12.017602 16418 raft_consensus.cc:348] T 33ec1c829bd9460a8cdcd383050518c2 P 11f96b80805a4844b6650c59f3d524eb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f96b80805a4844b6650c59f3d524eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41203 } } peers { permanent_uuid: "b08af7bd450046428e0a48e3edbcb12f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40201 } } peers { permanent_uuid: "699d9f5435bf4e2692984cf3bf4acd8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45283 } }
I20260925 04:42:12.018074 16418 raft_consensus.cc:729] T 33ec1c829bd9460a8cdcd383050518c2 P 11f96b80805a4844b6650c59f3d524eb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11f96b80805a4844b6650c59f3d524eb, State: Initialized, Role: FOLLOWER
I20260925 04:42:12.018540 16418 consensus_queue.cc:260] T 33ec1c829bd9460a8cdcd383050518c2 P 11f96b80805a4844b6650c59f3d524eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f96b80805a4844b6650c59f3d524eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41203 } } peers { permanent_uuid: "b08af7bd450046428e0a48e3edbcb12f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40201 } } peers { permanent_uuid: "699d9f5435bf4e2692984cf3bf4acd8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45283 } }
I20260925 04:42:12.019574 16418 ts_tablet_manager.cc:1428] T 33ec1c829bd9460a8cdcd383050518c2 P 11f96b80805a4844b6650c59f3d524eb: Time spent starting tablet: real 0.004s user 0.007s sys 0.000s
I20260925 04:42:12.019711 16526 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38799
I20260925 04:42:12.019953 16526 heartbeater.cc:464] Registering TS with master...
I20260925 04:42:12.020516 16526 heartbeater.cc:511] Master 127.0.98.62:38799 requested a full tablet report, sending...
I20260925 04:42:12.023307 16338 ts_manager.cc:194] Registered new tserver with Master: 11f96b80805a4844b6650c59f3d524eb (127.0.98.1:41203)
I20260925 04:42:12.026861 16338 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54167
I20260925 04:42:12.029608 16526 heartbeater.cc:503] Master 127.0.98.62:38799 was elected leader, sending a full tablet report...
Sep 25 04:42:12 dist-test-slave-dc95 krb5kdc[15756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311331, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 04:42:12.037981 16534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:12.026433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51718 (local address 127.0.98.1:41203)
0925 04:42:12.026846 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:12.026858 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:12.027412 (+ 554us) server_negotiation.cc:407] Connection header received
0925 04:42:12.027778 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:12.027784 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:12.028008 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:12.028143 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:42:12.028906 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:12.029998 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:12.031141 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:12.031673 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:12.031773 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:12.034339 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:42:12.034348 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:42:12.034359 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:42:12.034385 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:42:12.036041 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:42:12.036629 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:42:12.036634 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:42:12.036636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:42:12.036716 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:42:12.036985 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:42:12.036987 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:42:12.036988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:42:12.037234 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:42:12.037330 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:12.037686 (+ 356us) server_negotiation.cc:299] Negotiation successful
0925 04:42:12.037826 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":86,"threads_started":1}
I20260925 04:42:12.038002 16533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:12.026316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41203 (local address 127.0.0.1:51718)
0925 04:42:12.027228 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:12.027283 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 04:42:12.027498 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:42:12.028172 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:42:12.028178 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:42:12.028628 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:42:12.028764 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:12.028773 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:12.030134 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:12.030138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:12.031023 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:12.031033 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:12.031768 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:12.031784 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:12.031877 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:12.032428 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:42:12.032439 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:42:12.034188 (+ 1749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:42:12.036164 (+ 1976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:42:12.036176 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:42:12.036182 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:42:12.036515 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:42:12.036796 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:42:12.036798 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:42:12.036800 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:42:12.036893 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:42:12.037329 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:42:12.037335 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:42:12.037477 (+ 142us) client_negotiation.cc:769] Sending connection context
0925 04:42:12.037611 (+ 134us) client_negotiation.cc:240] Negotiation successful
0925 04:42:12.037822 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":129,"threads_started":1}
I20260925 04:42:12.039072 16460 tablet_service.cc:1514] Processing DeleteTablet for tablet 33ec1c829bd9460a8cdcd383050518c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:41:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51718
I20260925 04:42:12.039968 16535 tablet_replica.cc:330] stopping tablet replica
I20260925 04:42:12.040262 16535 raft_consensus.cc:2229] T 33ec1c829bd9460a8cdcd383050518c2 P 11f96b80805a4844b6650c59f3d524eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:42:12.040504 16535 raft_consensus.cc:2258] T 33ec1c829bd9460a8cdcd383050518c2 P 11f96b80805a4844b6650c59f3d524eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:42:12.041445 16535 ts_tablet_manager.cc:1905] T 33ec1c829bd9460a8cdcd383050518c2 P 11f96b80805a4844b6650c59f3d524eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:42:12.045305 16535 ts_tablet_manager.cc:1918] T 33ec1c829bd9460a8cdcd383050518c2 P 11f96b80805a4844b6650c59f3d524eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:42:12.045492 16535 log.cc:1196] T 33ec1c829bd9460a8cdcd383050518c2 P 11f96b80805a4844b6650c59f3d524eb: Deleting WAL directory at /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-0/wal/wals/33ec1c829bd9460a8cdcd383050518c2
I20260925 04:42:12.045742 16535 ts_tablet_manager.cc:1939] T 33ec1c829bd9460a8cdcd383050518c2 P 11f96b80805a4844b6650c59f3d524eb: Deleting consensus metadata
I20260925 04:42:12.047865 16324 catalog_manager.cc:4841] TS 11f96b80805a4844b6650c59f3d524eb (127.0.98.1:41203): tablet 33ec1c829bd9460a8cdcd383050518c2 (table test-table [id=00800117a2c6455cab6ed667fee91362]) successfully deleted
W20260925 04:42:12.133045 16531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:42:12.133365 16531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:42:12.133440 16531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:42:12.137773 16531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:42:12.137853 16531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:42:12.137941 16531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 04:42:12.143503 16531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:45283
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=46187
--tserver_master_addrs=127.0.98.62:38799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:42:12.144665 16531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:42:12.145841 16531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:42:12.156538 16540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:42:12.156543 16541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:42:12.156595 16531 server_base.cc:1034] running on GCE node
W20260925 04:42:12.156538 16543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:42:12.157182 16531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:42:12.157879 16531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:42:12.159050 16531 hybrid_clock.cc:648] HybridClock initialized: now 1790311332159018 us; error 45 us; skew 500 ppm
Sep 25 04:42:12 dist-test-slave-dc95 krb5kdc[15756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311332, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:42:12.161989 16531 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 04:42:12.163008 16531 webserver.cc:460] Webserver started at http://127.0.98.2:46187/ using document root <none> and password file <none>
I20260925 04:42:12.163591 16531 fs_manager.cc:359] Metadata directory not provided
I20260925 04:42:12.163723 16531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:42:12.168107 16531 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260925 04:42:12.170670 16550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:12.172072 16531 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:42:12.172190 16531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-1/wal
uuid: "699d9f5435bf4e2692984cf3bf4acd8c"
format_stamp: "Formatted at 2026-09-25 04:41:51 on dist-test-slave-dc95"
server_key: "5b61352dbd08d2fa97c18084f4038d15"
server_key_iv: "ad94de5435669c9804536b871efed19f"
server_key_version: "encryptionkey@0"
I20260925 04:42:12.172789 16531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:42:12.189961 16531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:42:12.193578 16531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:42:12.193764 16531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:42:12.194397 16531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:42:12.195504 16557 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 04:42:12.199877 16531 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 04:42:12.199965 16531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260925 04:42:12.200027 16531 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 04:42:12.202107 16531 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 04:42:12.202167 16531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260925 04:42:12.202487 16557 tablet_bootstrap.cc:492] T 33ec1c829bd9460a8cdcd383050518c2 P 699d9f5435bf4e2692984cf3bf4acd8c: Bootstrap starting.
I20260925 04:42:12.210057 16557 log.cc:824] T 33ec1c829bd9460a8cdcd383050518c2 P 699d9f5435bf4e2692984cf3bf4acd8c: Log is configured to *not* fsync() on all Append() calls
I20260925 04:42:12.211000 16531 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45283
I20260925 04:42:12.211301 16665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45283 every 8 connection(s)
I20260925 04:42:12.212001 16531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
I20260925 04:42:12.212874 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 16531
I20260925 04:42:12.213353 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:40201
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:38799
--webserver_port=41915
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:42:12.219133 16557 tablet_bootstrap.cc:492] T 33ec1c829bd9460a8cdcd383050518c2 P 699d9f5435bf4e2692984cf3bf4acd8c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 04:42:12.219650 16557 tablet_bootstrap.cc:492] T 33ec1c829bd9460a8cdcd383050518c2 P 699d9f5435bf4e2692984cf3bf4acd8c: Bootstrap complete.
I20260925 04:42:12.220052 16557 ts_tablet_manager.cc:1397] T 33ec1c829bd9460a8cdcd383050518c2 P 699d9f5435bf4e2692984cf3bf4acd8c: Time spent bootstrapping tablet: real 0.018s user 0.014s sys 0.003s
I20260925 04:42:12.222980 16557 raft_consensus.cc:348] T 33ec1c829bd9460a8cdcd383050518c2 P 699d9f5435bf4e2692984cf3bf4acd8c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f96b80805a4844b6650c59f3d524eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41203 } } peers { permanent_uuid: "b08af7bd450046428e0a48e3edbcb12f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40201 } } peers { permanent_uuid: "699d9f5435bf4e2692984cf3bf4acd8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45283 } }
I20260925 04:42:12.223444 16557 raft_consensus.cc:729] T 33ec1c829bd9460a8cdcd383050518c2 P 699d9f5435bf4e2692984cf3bf4acd8c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 699d9f5435bf4e2692984cf3bf4acd8c, State: Initialized, Role: FOLLOWER
I20260925 04:42:12.224004 16557 consensus_queue.cc:260] T 33ec1c829bd9460a8cdcd383050518c2 P 699d9f5435bf4e2692984cf3bf4acd8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f96b80805a4844b6650c59f3d524eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41203 } } peers { permanent_uuid: "b08af7bd450046428e0a48e3edbcb12f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40201 } } peers { permanent_uuid: "699d9f5435bf4e2692984cf3bf4acd8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45283 } }
I20260925 04:42:12.225023 16557 ts_tablet_manager.cc:1428] T 33ec1c829bd9460a8cdcd383050518c2 P 699d9f5435bf4e2692984cf3bf4acd8c: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
Sep 25 04:42:12 dist-test-slave-dc95 krb5kdc[15756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311332, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:42:12.230331 16528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:12.214820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55537 (local address 127.0.98.62:38799)
0925 04:42:12.215075 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:12.215079 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:12.215844 (+ 765us) server_negotiation.cc:407] Connection header received
0925 04:42:12.217709 (+ 1865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:12.217715 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:12.217784 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:12.217847 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:42:12.219764 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:12.220571 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:12.222390 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:12.222740 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:12.222870 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:12.226178 (+ 3308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:42:12.226181 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:42:12.226184 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:42:12.226216 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:42:12.227658 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:42:12.228562 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:42:12.228567 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:42:12.228569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:42:12.228682 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:42:12.229120 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:42:12.229124 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:42:12.229126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:42:12.229340 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:42:12.229445 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:12.230003 (+ 558us) server_negotiation.cc:299] Negotiation successful
0925 04:42:12.230156 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260925 04:42:12.231170 16669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:12.215144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38799 (local address 127.0.98.2:55537)
0925 04:42:12.215676 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:12.215731 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 04:42:12.217388 (+ 1657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:42:12.218293 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:42:12.218304 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:42:12.218815 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:42:12.219527 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:12.219550 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:12.220768 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:12.220772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:12.222222 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:12.222236 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:12.222894 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:12.222898 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:12.223125 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:12.224187 (+ 1062us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:42:12.224209 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:42:12.225982 (+ 1773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:42:12.227873 (+ 1891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:42:12.227882 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:42:12.227902 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:42:12.228409 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:42:12.228813 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:42:12.228816 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:42:12.228818 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:42:12.229004 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:42:12.229455 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:42:12.229462 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:42:12.229743 (+ 281us) client_negotiation.cc:769] Sending connection context
0925 04:42:12.229921 (+ 178us) client_negotiation.cc:240] Negotiation successful
0925 04:42:12.230272 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":116,"threads_started":1}
I20260925 04:42:12.232508 16666 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38799
I20260925 04:42:12.232758 16666 heartbeater.cc:464] Registering TS with master...
I20260925 04:42:12.233309 16666 heartbeater.cc:511] Master 127.0.98.62:38799 requested a full tablet report, sending...
I20260925 04:42:12.235052 16338 ts_manager.cc:194] Registered new tserver with Master: 699d9f5435bf4e2692984cf3bf4acd8c (127.0.98.2:45283)
I20260925 04:42:12.236261 16338 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55537
I20260925 04:42:12.238557 16666 heartbeater.cc:503] Master 127.0.98.62:38799 was elected leader, sending a full tablet report...
Sep 25 04:42:12 dist-test-slave-dc95 krb5kdc[15756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311331, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260925 04:42:12.245899 16533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:12.235951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45283 (local address 127.0.0.1:55342)
0925 04:42:12.236128 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:12.236142 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:42:12.236218 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:42:12.236933 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:42:12.236937 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:42:12.237140 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:42:12.237252 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:12.237260 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:12.238571 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:12.238575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:12.239379 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:12.239392 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:12.240178 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:12.240195 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:12.240298 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:12.240771 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:42:12.240786 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:42:12.242164 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:42:12.244195 (+ 2031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:42:12.244199 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:42:12.244201 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:42:12.244419 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:42:12.244772 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:42:12.244776 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:42:12.244778 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:42:12.244869 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:42:12.245350 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:42:12.245353 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:42:12.245458 (+ 105us) client_negotiation.cc:769] Sending connection context
0925 04:42:12.245548 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 04:42:12.245718 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106}
I20260925 04:42:12.245980 16671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:12.236086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55342 (local address 127.0.98.2:45283)
0925 04:42:12.236469 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:12.236481 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:12.236505 (+ 24us) server_negotiation.cc:407] Connection header received
0925 04:42:12.236572 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:12.236578 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:12.236789 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:12.236925 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:42:12.237380 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:12.238434 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:12.239511 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:12.240083 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:12.240168 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:12.242313 (+ 2145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:42:12.242325 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:42:12.242337 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:42:12.242362 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:42:12.244067 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:42:12.244531 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:42:12.244536 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:42:12.244537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:42:12.244633 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:42:12.244954 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:42:12.244958 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:42:12.244960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:42:12.245221 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:42:12.245355 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:12.245644 (+ 289us) server_negotiation.cc:299] Negotiation successful
0925 04:42:12.245822 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":284,"thread_start_us":123,"threads_started":1}
I20260925 04:42:12.246981 16584 tablet_service.cc:1514] Processing DeleteTablet for tablet 33ec1c829bd9460a8cdcd383050518c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:41:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55342
I20260925 04:42:12.247988 16672 tablet_replica.cc:330] stopping tablet replica
I20260925 04:42:12.248353 16672 raft_consensus.cc:2229] T 33ec1c829bd9460a8cdcd383050518c2 P 699d9f5435bf4e2692984cf3bf4acd8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:42:12.248633 16672 raft_consensus.cc:2258] T 33ec1c829bd9460a8cdcd383050518c2 P 699d9f5435bf4e2692984cf3bf4acd8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:42:12.249818 16672 ts_tablet_manager.cc:1905] T 33ec1c829bd9460a8cdcd383050518c2 P 699d9f5435bf4e2692984cf3bf4acd8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:42:12.254793 16672 ts_tablet_manager.cc:1918] T 33ec1c829bd9460a8cdcd383050518c2 P 699d9f5435bf4e2692984cf3bf4acd8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:42:12.254923 16672 log.cc:1196] T 33ec1c829bd9460a8cdcd383050518c2 P 699d9f5435bf4e2692984cf3bf4acd8c: Deleting WAL directory at /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-1/wal/wals/33ec1c829bd9460a8cdcd383050518c2
I20260925 04:42:12.255180 16672 ts_tablet_manager.cc:1939] T 33ec1c829bd9460a8cdcd383050518c2 P 699d9f5435bf4e2692984cf3bf4acd8c: Deleting consensus metadata
I20260925 04:42:12.257225 16325 catalog_manager.cc:4841] TS 699d9f5435bf4e2692984cf3bf4acd8c (127.0.98.2:45283): tablet 33ec1c829bd9460a8cdcd383050518c2 (table test-table [id=00800117a2c6455cab6ed667fee91362]) successfully deleted
W20260925 04:42:12.335966 16668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:42:12.336242 16668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:42:12.336285 16668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:42:12.340812 16668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:42:12.340903 16668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:42:12.340971 16668 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 04:42:12.346891 16668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:40201
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=41915
--tserver_master_addrs=127.0.98.62:38799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:42:12.348064 16668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:42:12.349083 16668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:42:12.359306 16677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:42:12.359315 16680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:42:12.359488 16668 server_base.cc:1034] running on GCE node
W20260925 04:42:12.359330 16678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:42:12.360193 16668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:42:12.360929 16668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:42:12.362107 16668 hybrid_clock.cc:648] HybridClock initialized: now 1790311332362076 us; error 44 us; skew 500 ppm
Sep 25 04:42:12 dist-test-slave-dc95 krb5kdc[15756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311332, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:42:12.365154 16668 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 04:42:12.366214 16668 webserver.cc:460] Webserver started at http://127.0.98.3:41915/ using document root <none> and password file <none>
I20260925 04:42:12.366775 16668 fs_manager.cc:359] Metadata directory not provided
I20260925 04:42:12.366855 16668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:42:12.371001 16668 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:42:12.373514 16687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:12.374891 16668 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:42:12.375033 16668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-2/wal
uuid: "b08af7bd450046428e0a48e3edbcb12f"
format_stamp: "Formatted at 2026-09-25 04:41:51 on dist-test-slave-dc95"
server_key: "680b0018850ddae6948b89e79157a822"
server_key_iv: "204bba1af1aca76b5183c55413db3960"
server_key_version: "encryptionkey@0"
I20260925 04:42:12.375663 16668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:42:12.408197 16668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:42:12.411650 16668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:42:12.411864 16668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:42:12.412459 16668 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:42:12.413399 16668 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:42:12.413448 16668 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:12.413489 16668 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:42:12.413511 16668 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:12.422077 16668 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40201
I20260925 04:42:12.422104 16800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40201 every 8 connection(s)
I20260925 04:42:12.423092 16668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
I20260925 04:42:12.431386 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 16668
Sep 25 04:42:12 dist-test-slave-dc95 krb5kdc[15756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311332, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:42:12.437590 16528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:12.424991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44329 (local address 127.0.98.62:38799)
0925 04:42:12.425110 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:12.425114 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:12.426201 (+ 1087us) server_negotiation.cc:407] Connection header received
0925 04:42:12.427102 (+ 901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:12.427106 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:12.427168 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:12.427247 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:42:12.428560 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:12.429095 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:12.430264 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:12.430495 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:12.430586 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:12.433502 (+ 2916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:42:12.433506 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:42:12.433509 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:42:12.433536 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:42:12.435088 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:42:12.435729 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:42:12.435732 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:42:12.435734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:42:12.435881 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:42:12.436309 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:42:12.436312 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:42:12.436314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:42:12.436562 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:42:12.436685 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:12.437316 (+ 631us) server_negotiation.cc:299] Negotiation successful
0925 04:42:12.437456 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42,"spinlock_wait_cycles":20224}
I20260925 04:42:12.438473 16803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:12.425358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38799 (local address 127.0.98.3:44329)
0925 04:42:12.426035 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:12.426077 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 04:42:12.426886 (+ 809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:42:12.427471 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:42:12.427479 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:42:12.427900 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:42:12.428403 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:12.428418 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:12.429229 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:12.429232 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:12.430127 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:12.430137 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:12.430646 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:12.430650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:12.430819 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:12.431449 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:42:12.431472 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:42:12.433275 (+ 1803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:42:12.435223 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:42:12.435229 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:42:12.435244 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:42:12.435638 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:42:12.435984 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:42:12.435986 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:42:12.435988 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:42:12.436157 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:42:12.436680 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:42:12.436686 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:42:12.436989 (+ 303us) client_negotiation.cc:769] Sending connection context
0925 04:42:12.437167 (+ 178us) client_negotiation.cc:240] Negotiation successful
0925 04:42:12.437543 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":119,"threads_started":1}
I20260925 04:42:12.439828 16801 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38799
I20260925 04:42:12.440119 16801 heartbeater.cc:464] Registering TS with master...
I20260925 04:42:12.440790 16801 heartbeater.cc:511] Master 127.0.98.62:38799 requested a full tablet report, sending...
I20260925 04:42:12.440817 16805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:12.432155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46946 (local address 127.0.98.62:38799)
0925 04:42:12.432411 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:12.432415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:12.432515 (+ 100us) server_negotiation.cc:407] Connection header received
0925 04:42:12.432658 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:12.432660 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:12.432701 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:12.432783 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:42:12.433541 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:12.434172 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:12.435359 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:12.435722 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:12.435804 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:12.437235 (+ 1431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:42:12.437239 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:42:12.437241 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:42:12.437265 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:42:12.438963 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:42:12.439490 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:42:12.439492 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:42:12.439494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:42:12.439570 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:42:12.439882 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:42:12.439885 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:42:12.439887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:42:12.440112 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:42:12.440325 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:12.440520 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 04:42:12.440653 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":71,"threads_started":1}
I20260925 04:42:12.441921 16338 ts_manager.cc:194] Registered new tserver with Master: b08af7bd450046428e0a48e3edbcb12f (127.0.98.3:40201)
I20260925 04:42:12.442724 16338 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44329
I20260925 04:42:12.442842 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:42:12.447302 16805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:12.443458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46950 (local address 127.0.98.62:38799)
0925 04:42:12.443587 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:12.443592 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:12.443803 (+ 211us) server_negotiation.cc:407] Connection header received
0925 04:42:12.443942 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:12.443944 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:12.444073 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:12.444188 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:42:12.444361 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:12.445179 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:12.446197 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:12.446469 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:12.446551 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:12.446849 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:42:12.446979 (+ 130us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0925 04:42:12.447190 (+ 211us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46950: token expired
Metrics: {"server-negotiator.queue_time_us":40}
W20260925 04:42:12.447405 16805 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46950: token expired
W20260925 04:42:12.447804 16266 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:38799)
I20260925 04:42:12.454929 16805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:12.448155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46964 (local address 127.0.98.62:38799)
0925 04:42:12.448269 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:12.448273 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:12.448286 (+ 13us) server_negotiation.cc:407] Connection header received
0925 04:42:12.448398 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:12.448401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:12.448444 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:12.448507 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:42:12.449097 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:12.449602 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:12.450389 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:12.450611 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:12.450698 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:12.452043 (+ 1345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:42:12.452047 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:42:12.452049 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:42:12.452080 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:42:12.453345 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:42:12.453745 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:42:12.453749 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:42:12.453751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:42:12.453845 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:42:12.454085 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:42:12.454088 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:42:12.454090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:42:12.454278 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:42:12.454398 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:12.454636 (+ 238us) server_negotiation.cc:299] Negotiation successful
0925 04:42:12.454759 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 04:42:12.457724 16338 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46964:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:42:12.461313 16338 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:42:12.469822 16584 tablet_service.cc:1467] Processing CreateTablet for tablet 314ff12e791e4ceb88d0bc5a81d3c6f9 (DEFAULT_TABLE table=test-table [id=2c556242594648c2a688ffc5ac88f122]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:42:12.469822 16460 tablet_service.cc:1467] Processing CreateTablet for tablet 314ff12e791e4ceb88d0bc5a81d3c6f9 (DEFAULT_TABLE table=test-table [id=2c556242594648c2a688ffc5ac88f122]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:42:12.470475 16584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 314ff12e791e4ceb88d0bc5a81d3c6f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:42:12.470476 16460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 314ff12e791e4ceb88d0bc5a81d3c6f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:42:12.473497 16418 tablet_bootstrap.cc:492] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 11f96b80805a4844b6650c59f3d524eb: Bootstrap starting.
I20260925 04:42:12.473497 16557 tablet_bootstrap.cc:492] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 699d9f5435bf4e2692984cf3bf4acd8c: Bootstrap starting.
I20260925 04:42:12.473832 16533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:12.468918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40201 (local address 127.0.0.1:42152)
0925 04:42:12.469046 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:12.469062 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:42:12.469164 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:42:12.469908 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:42:12.469911 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:42:12.469930 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:42:12.470037 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:12.470043 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:12.471550 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:12.471554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:12.472431 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:12.472443 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:12.473319 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:12.473323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:12.473422 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:12.473508 (+ 86us) client_negotiation.cc:769] Sending connection context
0925 04:42:12.473583 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 04:42:12.473688 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 04:42:12.474184 16810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:12.469103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42152 (local address 127.0.98.3:40201)
0925 04:42:12.469476 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:12.469486 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:12.469507 (+ 21us) server_negotiation.cc:407] Connection header received
0925 04:42:12.469573 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:12.469577 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:12.469760 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:12.469911 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:42:12.470131 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:12.471415 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:12.472561 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:12.473190 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:12.473295 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:12.473461 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:12.473724 (+ 263us) server_negotiation.cc:299] Negotiation successful
0925 04:42:12.473832 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":171,"threads_started":1}
I20260925 04:42:12.474453 16557 tablet_bootstrap.cc:654] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 699d9f5435bf4e2692984cf3bf4acd8c: Neither blocks nor log segments found. Creating new log.
I20260925 04:42:12.474453 16418 tablet_bootstrap.cc:654] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 11f96b80805a4844b6650c59f3d524eb: Neither blocks nor log segments found. Creating new log.
I20260925 04:42:12.476070 16418 tablet_bootstrap.cc:492] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 11f96b80805a4844b6650c59f3d524eb: No bootstrap required, opened a new log
I20260925 04:42:12.476177 16418 ts_tablet_manager.cc:1397] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 11f96b80805a4844b6650c59f3d524eb: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 04:42:12.476517 16557 tablet_bootstrap.cc:492] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 699d9f5435bf4e2692984cf3bf4acd8c: No bootstrap required, opened a new log
I20260925 04:42:12.476629 16557 ts_tablet_manager.cc:1397] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 699d9f5435bf4e2692984cf3bf4acd8c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 04:42:12.476785 16418 raft_consensus.cc:348] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 11f96b80805a4844b6650c59f3d524eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b08af7bd450046428e0a48e3edbcb12f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40201 } } peers { permanent_uuid: "699d9f5435bf4e2692984cf3bf4acd8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45283 } } peers { permanent_uuid: "11f96b80805a4844b6650c59f3d524eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41203 } }
I20260925 04:42:12.476946 16418 raft_consensus.cc:374] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 11f96b80805a4844b6650c59f3d524eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:42:12.476986 16418 raft_consensus.cc:729] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 11f96b80805a4844b6650c59f3d524eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11f96b80805a4844b6650c59f3d524eb, State: Initialized, Role: FOLLOWER
I20260925 04:42:12.477036 16557 raft_consensus.cc:348] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 699d9f5435bf4e2692984cf3bf4acd8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b08af7bd450046428e0a48e3edbcb12f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40201 } } peers { permanent_uuid: "699d9f5435bf4e2692984cf3bf4acd8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45283 } } peers { permanent_uuid: "11f96b80805a4844b6650c59f3d524eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41203 } }
I20260925 04:42:12.477151 16557 raft_consensus.cc:374] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 699d9f5435bf4e2692984cf3bf4acd8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:42:12.477190 16557 raft_consensus.cc:729] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 699d9f5435bf4e2692984cf3bf4acd8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 699d9f5435bf4e2692984cf3bf4acd8c, State: Initialized, Role: FOLLOWER
I20260925 04:42:12.477159 16418 consensus_queue.cc:260] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 11f96b80805a4844b6650c59f3d524eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b08af7bd450046428e0a48e3edbcb12f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40201 } } peers { permanent_uuid: "699d9f5435bf4e2692984cf3bf4acd8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45283 } } peers { permanent_uuid: "11f96b80805a4844b6650c59f3d524eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41203 } }
I20260925 04:42:12.477432 16418 ts_tablet_manager.cc:1428] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 11f96b80805a4844b6650c59f3d524eb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 04:42:12.477486 16557 consensus_queue.cc:260] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 699d9f5435bf4e2692984cf3bf4acd8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b08af7bd450046428e0a48e3edbcb12f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40201 } } peers { permanent_uuid: "699d9f5435bf4e2692984cf3bf4acd8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45283 } } peers { permanent_uuid: "11f96b80805a4844b6650c59f3d524eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41203 } }
I20260925 04:42:12.477751 16557 ts_tablet_manager.cc:1428] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 699d9f5435bf4e2692984cf3bf4acd8c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 04:42:12.477924 16735 tablet_service.cc:1467] Processing CreateTablet for tablet 314ff12e791e4ceb88d0bc5a81d3c6f9 (DEFAULT_TABLE table=test-table [id=2c556242594648c2a688ffc5ac88f122]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:42:12.479076 16735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 314ff12e791e4ceb88d0bc5a81d3c6f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:42:12.486944 16813 tablet_bootstrap.cc:492] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P b08af7bd450046428e0a48e3edbcb12f: Bootstrap starting.
I20260925 04:42:12.489607 16813 tablet_bootstrap.cc:654] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P b08af7bd450046428e0a48e3edbcb12f: Neither blocks nor log segments found. Creating new log.
I20260925 04:42:12.490387 16813 log.cc:824] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P b08af7bd450046428e0a48e3edbcb12f: Log is configured to *not* fsync() on all Append() calls
I20260925 04:42:12.492481 16813 tablet_bootstrap.cc:492] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P b08af7bd450046428e0a48e3edbcb12f: No bootstrap required, opened a new log
I20260925 04:42:12.492674 16813 ts_tablet_manager.cc:1397] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P b08af7bd450046428e0a48e3edbcb12f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 04:42:12.495940 16813 raft_consensus.cc:348] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P b08af7bd450046428e0a48e3edbcb12f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b08af7bd450046428e0a48e3edbcb12f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40201 } } peers { permanent_uuid: "699d9f5435bf4e2692984cf3bf4acd8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45283 } } peers { permanent_uuid: "11f96b80805a4844b6650c59f3d524eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41203 } }
I20260925 04:42:12.496236 16813 raft_consensus.cc:374] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P b08af7bd450046428e0a48e3edbcb12f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:42:12.496272 16813 raft_consensus.cc:729] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P b08af7bd450046428e0a48e3edbcb12f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b08af7bd450046428e0a48e3edbcb12f, State: Initialized, Role: FOLLOWER
W20260925 04:42:12.496704 16527 tablet.cc:2369] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 11f96b80805a4844b6650c59f3d524eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:42:12.496699 16813 consensus_queue.cc:260] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P b08af7bd450046428e0a48e3edbcb12f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b08af7bd450046428e0a48e3edbcb12f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40201 } } peers { permanent_uuid: "699d9f5435bf4e2692984cf3bf4acd8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45283 } } peers { permanent_uuid: "11f96b80805a4844b6650c59f3d524eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41203 } }
I20260925 04:42:12.497474 16801 heartbeater.cc:503] Master 127.0.98.62:38799 was elected leader, sending a full tablet report...
I20260925 04:42:12.497648 16813 ts_tablet_manager.cc:1428] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P b08af7bd450046428e0a48e3edbcb12f: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 04:42:12.525880 16815 raft_consensus.cc:482] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P b08af7bd450046428e0a48e3edbcb12f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:42:12.526114 16815 raft_consensus.cc:504] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P b08af7bd450046428e0a48e3edbcb12f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b08af7bd450046428e0a48e3edbcb12f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40201 } } peers { permanent_uuid: "699d9f5435bf4e2692984cf3bf4acd8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45283 } } peers { permanent_uuid: "11f96b80805a4844b6650c59f3d524eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41203 } }
I20260925 04:42:12.527395 16815 leader_election.cc:288] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P b08af7bd450046428e0a48e3edbcb12f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 699d9f5435bf4e2692984cf3bf4acd8c (127.0.98.2:45283), 11f96b80805a4844b6650c59f3d524eb (127.0.98.1:41203)
I20260925 04:42:12.531378 16534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:12.527764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38105 (local address 127.0.98.1:41203)
0925 04:42:12.527882 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:12.527886 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:12.528246 (+ 360us) server_negotiation.cc:407] Connection header received
0925 04:42:12.528331 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:12.528334 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:12.528397 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:12.528501 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:42:12.528739 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:12.529232 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:12.530358 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:12.530821 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:12.530918 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:12.531071 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:12.531161 (+ 90us) server_negotiation.cc:299] Negotiation successful
0925 04:42:12.531238 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 04:42:12.531378 16803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:12.527672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45283 (local address 127.0.98.3:41717)
0925 04:42:12.527792 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:12.527816 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 04:42:12.527907 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:42:12.528318 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:42:12.528321 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:42:12.528342 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:42:12.528472 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:12.528480 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:12.529324 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:12.529327 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:12.530217 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:12.530227 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:12.530928 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:12.530931 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:12.531033 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:12.531055 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 04:42:12.531136 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 04:42:12.531233 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":70}
I20260925 04:42:12.531395 16671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:12.527764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41717 (local address 127.0.98.2:45283)
0925 04:42:12.527882 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:12.527886 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:12.527901 (+ 15us) server_negotiation.cc:407] Connection header received
0925 04:42:12.528017 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:12.528021 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:12.528087 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:12.528213 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:42:12.528581 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:12.529196 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:12.530356 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:12.530803 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:12.530893 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:12.531069 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:12.531157 (+ 88us) server_negotiation.cc:299] Negotiation successful
0925 04:42:12.531237 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 04:42:12.531395 16816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:12.527672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41203 (local address 127.0.98.3:38105)
0925 04:42:12.528121 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:12.528135 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:42:12.528245 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:42:12.528510 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:42:12.528512 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:42:12.528526 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:42:12.528640 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:12.528646 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:12.529343 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:12.529346 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:12.530228 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:12.530237 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:12.530942 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:12.530946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:12.531042 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:12.531064 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 04:42:12.531133 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 04:42:12.531244 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"mutex_wait_us":157,"thread_start_us":123,"threads_started":1}
I20260925 04:42:12.532196 16605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "314ff12e791e4ceb88d0bc5a81d3c6f9" candidate_uuid: "b08af7bd450046428e0a48e3edbcb12f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "699d9f5435bf4e2692984cf3bf4acd8c" is_pre_election: true
I20260925 04:42:12.532194 16480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "314ff12e791e4ceb88d0bc5a81d3c6f9" candidate_uuid: "b08af7bd450046428e0a48e3edbcb12f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11f96b80805a4844b6650c59f3d524eb" is_pre_election: true
I20260925 04:42:12.532550 16480 raft_consensus.cc:2454] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 11f96b80805a4844b6650c59f3d524eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b08af7bd450046428e0a48e3edbcb12f in term 0.
I20260925 04:42:12.532550 16605 raft_consensus.cc:2454] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 699d9f5435bf4e2692984cf3bf4acd8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b08af7bd450046428e0a48e3edbcb12f in term 0.
I20260925 04:42:12.533069 16690 leader_election.cc:302] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P b08af7bd450046428e0a48e3edbcb12f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 11f96b80805a4844b6650c59f3d524eb, 699d9f5435bf4e2692984cf3bf4acd8c, b08af7bd450046428e0a48e3edbcb12f; no voters:
I20260925 04:42:12.533442 16815 raft_consensus.cc:2781] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P b08af7bd450046428e0a48e3edbcb12f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:42:12.533524 16815 raft_consensus.cc:482] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P b08af7bd450046428e0a48e3edbcb12f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:42:12.533553 16815 raft_consensus.cc:3037] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P b08af7bd450046428e0a48e3edbcb12f [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:42:12.534693 16815 raft_consensus.cc:504] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P b08af7bd450046428e0a48e3edbcb12f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b08af7bd450046428e0a48e3edbcb12f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40201 } } peers { permanent_uuid: "699d9f5435bf4e2692984cf3bf4acd8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45283 } } peers { permanent_uuid: "11f96b80805a4844b6650c59f3d524eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41203 } }
I20260925 04:42:12.535070 16815 leader_election.cc:288] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P b08af7bd450046428e0a48e3edbcb12f [CANDIDATE]: Term 1 election: Requested vote from peers 699d9f5435bf4e2692984cf3bf4acd8c (127.0.98.2:45283), 11f96b80805a4844b6650c59f3d524eb (127.0.98.1:41203)
I20260925 04:42:12.535549 16605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "314ff12e791e4ceb88d0bc5a81d3c6f9" candidate_uuid: "b08af7bd450046428e0a48e3edbcb12f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "699d9f5435bf4e2692984cf3bf4acd8c"
I20260925 04:42:12.535549 16480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "314ff12e791e4ceb88d0bc5a81d3c6f9" candidate_uuid: "b08af7bd450046428e0a48e3edbcb12f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11f96b80805a4844b6650c59f3d524eb"
I20260925 04:42:12.535737 16480 raft_consensus.cc:3037] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 11f96b80805a4844b6650c59f3d524eb [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:42:12.535737 16605 raft_consensus.cc:3037] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 699d9f5435bf4e2692984cf3bf4acd8c [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:42:12.537077 16605 raft_consensus.cc:2454] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 699d9f5435bf4e2692984cf3bf4acd8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b08af7bd450046428e0a48e3edbcb12f in term 1.
I20260925 04:42:12.537077 16480 raft_consensus.cc:2454] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 11f96b80805a4844b6650c59f3d524eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b08af7bd450046428e0a48e3edbcb12f in term 1.
I20260925 04:42:12.537505 16690 leader_election.cc:302] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P b08af7bd450046428e0a48e3edbcb12f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11f96b80805a4844b6650c59f3d524eb, b08af7bd450046428e0a48e3edbcb12f; no voters:
I20260925 04:42:12.537720 16815 raft_consensus.cc:2781] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P b08af7bd450046428e0a48e3edbcb12f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:42:12.537953 16815 raft_consensus.cc:686] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P b08af7bd450046428e0a48e3edbcb12f [term 1 LEADER]: Becoming Leader. State: Replica: b08af7bd450046428e0a48e3edbcb12f, State: Running, Role: LEADER
I20260925 04:42:12.538302 16815 consensus_queue.cc:237] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P b08af7bd450046428e0a48e3edbcb12f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b08af7bd450046428e0a48e3edbcb12f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40201 } } peers { permanent_uuid: "699d9f5435bf4e2692984cf3bf4acd8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45283 } } peers { permanent_uuid: "11f96b80805a4844b6650c59f3d524eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41203 } }
I20260925 04:42:12.541970 16337 catalog_manager.cc:5495] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P b08af7bd450046428e0a48e3edbcb12f reported cstate change: term changed from 0 to 1, leader changed from <none> to b08af7bd450046428e0a48e3edbcb12f (127.0.98.3). New cstate: current_term: 1 leader_uuid: "b08af7bd450046428e0a48e3edbcb12f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b08af7bd450046428e0a48e3edbcb12f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40201 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "699d9f5435bf4e2692984cf3bf4acd8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11f96b80805a4844b6650c59f3d524eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41203 } health_report { overall_health: UNKNOWN } } }
I20260925 04:42:12.564538 16810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:12.560950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42166 (local address 127.0.98.3:40201)
0925 04:42:12.561080 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:12.561084 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:12.561097 (+ 13us) server_negotiation.cc:407] Connection header received
0925 04:42:12.561174 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:12.561176 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:12.561256 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:12.561345 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:42:12.561561 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:12.562053 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:12.562936 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:12.563171 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:12.563301 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:12.563625 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:42:12.564106 (+ 481us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:42:12.564198 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:12.564286 (+ 88us) server_negotiation.cc:299] Negotiation successful
0925 04:42:12.564350 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 04:42:12.570801 16480 raft_consensus.cc:1261] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 11f96b80805a4844b6650c59f3d524eb [term 1 FOLLOWER]: Refusing update from remote peer b08af7bd450046428e0a48e3edbcb12f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:42:12.570801 16605 raft_consensus.cc:1261] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 699d9f5435bf4e2692984cf3bf4acd8c [term 1 FOLLOWER]: Refusing update from remote peer b08af7bd450046428e0a48e3edbcb12f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:42:12.571521 16817 consensus_queue.cc:1035] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P b08af7bd450046428e0a48e3edbcb12f [LEADER]: Connected to new peer: Peer: permanent_uuid: "699d9f5435bf4e2692984cf3bf4acd8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:42:12.571738 16815 consensus_queue.cc:1035] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P b08af7bd450046428e0a48e3edbcb12f [LEADER]: Connected to new peer: Peer: permanent_uuid: "11f96b80805a4844b6650c59f3d524eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:42:12.588477 16337 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:42:12.588685 16337 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:42:12.591383 16337 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7fc2639fe7ca454c8de7a2cfce0e63c3: Sending DeleteTablet for 3 replicas of tablet 314ff12e791e4ceb88d0bc5a81d3c6f9
I20260925 04:42:12.592276 16460 tablet_service.cc:1514] Processing DeleteTablet for tablet 314ff12e791e4ceb88d0bc5a81d3c6f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:42:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51718
I20260925 04:42:12.592386 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 16376
I20260925 04:42:12.592941 16584 tablet_service.cc:1514] Processing DeleteTablet for tablet 314ff12e791e4ceb88d0bc5a81d3c6f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:42:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55342
I20260925 04:42:12.593189 16735 tablet_service.cc:1514] Processing DeleteTablet for tablet 314ff12e791e4ceb88d0bc5a81d3c6f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:42:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42152
I20260925 04:42:12.593688 16830 tablet_replica.cc:330] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P b08af7bd450046428e0a48e3edbcb12f: stopping tablet replica
I20260925 04:42:12.593709 16672 tablet_replica.cc:330] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 699d9f5435bf4e2692984cf3bf4acd8c: stopping tablet replica
I20260925 04:42:12.593834 16672 raft_consensus.cc:2229] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 699d9f5435bf4e2692984cf3bf4acd8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:42:12.593990 16672 raft_consensus.cc:2258] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 699d9f5435bf4e2692984cf3bf4acd8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:42:12.594059 16830 raft_consensus.cc:2229] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P b08af7bd450046428e0a48e3edbcb12f [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:42:12.594506 16830 raft_consensus.cc:2258] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P b08af7bd450046428e0a48e3edbcb12f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:42:12.595834 16672 ts_tablet_manager.cc:1905] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 699d9f5435bf4e2692984cf3bf4acd8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:42:12.596181 16830 ts_tablet_manager.cc:1905] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P b08af7bd450046428e0a48e3edbcb12f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:42:12.598531 16672 ts_tablet_manager.cc:1918] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 699d9f5435bf4e2692984cf3bf4acd8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:42:12.598626 16672 log.cc:1196] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 699d9f5435bf4e2692984cf3bf4acd8c: Deleting WAL directory at /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-1/wal/wals/314ff12e791e4ceb88d0bc5a81d3c6f9
I20260925 04:42:12.598881 16672 ts_tablet_manager.cc:1939] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 699d9f5435bf4e2692984cf3bf4acd8c: Deleting consensus metadata
I20260925 04:42:12.598914 16830 ts_tablet_manager.cc:1918] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P b08af7bd450046428e0a48e3edbcb12f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:42:12.599014 16830 log.cc:1196] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P b08af7bd450046428e0a48e3edbcb12f: Deleting WAL directory at /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-2/wal/wals/314ff12e791e4ceb88d0bc5a81d3c6f9
I20260925 04:42:12.599324 16830 ts_tablet_manager.cc:1939] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P b08af7bd450046428e0a48e3edbcb12f: Deleting consensus metadata
W20260925 04:42:12.599942 16324 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41203 (error 108)
I20260925 04:42:12.599957 16325 catalog_manager.cc:4841] TS 699d9f5435bf4e2692984cf3bf4acd8c (127.0.98.2:45283): tablet 314ff12e791e4ceb88d0bc5a81d3c6f9 (table test-table [id=2c556242594648c2a688ffc5ac88f122]) successfully deleted
I20260925 04:42:12.600582 16325 catalog_manager.cc:4841] TS b08af7bd450046428e0a48e3edbcb12f (127.0.98.3:40201): tablet 314ff12e791e4ceb88d0bc5a81d3c6f9 (table test-table [id=2c556242594648c2a688ffc5ac88f122]) successfully deleted
W20260925 04:42:12.601071 16533 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:42:12.600757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41203 (local address 127.0.0.1:51734)
0925 04:42:12.600921 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:12.600988 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41203: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":92}
I20260925 04:42:12.601294 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 16531
W20260925 04:42:12.601230 16324 catalog_manager.cc:4584] TS 11f96b80805a4844b6650c59f3d524eb (127.0.98.1:41203): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 314ff12e791e4ceb88d0bc5a81d3c6f9: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41203: connect: Connection reset by peer (error 104)
I20260925 04:42:12.608700 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 16668
I20260925 04:42:12.614219 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 16305
I20260925 04:42:32.620491 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38799
--webserver_interface=127.0.98.62
--webserver_port=43981
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38799
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:42:32.737434 16832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:42:32.737735 16832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:42:32.737782 16832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:42:32.742005 16832 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 04:42:32.742089 16832 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 04:42:32.742103 16832 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 04:42:32.742115 16832 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:42:32.742133 16832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:42:32.742151 16832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:42:32.742169 16832 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:42:32.742197 16832 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:42:32.748015 16832 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=43981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:42:32.749238 16832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:42:32.750669 16832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:42:32.760954 16837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:42:32.761000 16832 server_base.cc:1034] running on GCE node
W20260925 04:42:32.760947 16838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:42:32.760941 16840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:42:32.761880 16832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:42:32.763129 16832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:42:32.764358 16832 hybrid_clock.cc:648] HybridClock initialized: now 1790311352764333 us; error 43 us; skew 500 ppm
Sep 25 04:42:32 dist-test-slave-dc95 krb5kdc[15756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311352, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:42:32.767381 16832 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 04:42:32.768404 16832 webserver.cc:460] Webserver started at http://127.0.98.62:43981/ using document root <none> and password file <none>
I20260925 04:42:32.768962 16832 fs_manager.cc:359] Metadata directory not provided
I20260925 04:42:32.769034 16832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:42:32.773255 16832 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260925 04:42:32.775751 16847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:32.777139 16832 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:42:32.777300 16832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/master-0/wal
uuid: "7fc2639fe7ca454c8de7a2cfce0e63c3"
format_stamp: "Formatted at 2026-09-25 04:41:50 on dist-test-slave-dc95"
server_key: "7541e435ee78e02d5809b5333f7ccd0c"
server_key_iv: "8dfe44d99c78a72e7614d359866b216c"
server_key_version: "encryptionkey@0"
I20260925 04:42:32.777892 16832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:42:32.792603 16832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:42:32.796052 16832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:42:32.796262 16832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:42:32.803702 16832 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38799
I20260925 04:42:32.804103 16899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38799 every 8 connection(s)
I20260925 04:42:32.804857 16832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
I20260925 04:42:32.807150 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 16832
I20260925 04:42:32.807580 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41203
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:38799
--webserver_port=35771
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:42:32.809624 16900 sys_catalog.cc:263] Verifying existing consensus state
I20260925 04:42:32.811492 16900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fc2639fe7ca454c8de7a2cfce0e63c3: Bootstrap starting.
I20260925 04:42:32.829202 16900 log.cc:824] T 00000000000000000000000000000000 P 7fc2639fe7ca454c8de7a2cfce0e63c3: Log is configured to *not* fsync() on all Append() calls
I20260925 04:42:32.841653 16900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fc2639fe7ca454c8de7a2cfce0e63c3: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 04:42:32.842087 16900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fc2639fe7ca454c8de7a2cfce0e63c3: Bootstrap complete.
I20260925 04:42:32.846160 16900 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7fc2639fe7ca454c8de7a2cfce0e63c3 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fc2639fe7ca454c8de7a2cfce0e63c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38799 } }
I20260925 04:42:32.846632 16900 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7fc2639fe7ca454c8de7a2cfce0e63c3 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fc2639fe7ca454c8de7a2cfce0e63c3, State: Initialized, Role: FOLLOWER
I20260925 04:42:32.847159 16900 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7fc2639fe7ca454c8de7a2cfce0e63c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fc2639fe7ca454c8de7a2cfce0e63c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38799 } }
I20260925 04:42:32.847342 16900 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7fc2639fe7ca454c8de7a2cfce0e63c3 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:42:32.847415 16900 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7fc2639fe7ca454c8de7a2cfce0e63c3 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:42:32.847522 16900 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7fc2639fe7ca454c8de7a2cfce0e63c3 [term 2 FOLLOWER]: Advancing to term 3
I20260925 04:42:32.849635 16900 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7fc2639fe7ca454c8de7a2cfce0e63c3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fc2639fe7ca454c8de7a2cfce0e63c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38799 } }
I20260925 04:42:32.850014 16900 leader_election.cc:302] T 00000000000000000000000000000000 P 7fc2639fe7ca454c8de7a2cfce0e63c3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7fc2639fe7ca454c8de7a2cfce0e63c3; no voters:
I20260925 04:42:32.850364 16900 leader_election.cc:288] T 00000000000000000000000000000000 P 7fc2639fe7ca454c8de7a2cfce0e63c3 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 04:42:32.850534 16904 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7fc2639fe7ca454c8de7a2cfce0e63c3 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 04:42:32.850929 16904 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7fc2639fe7ca454c8de7a2cfce0e63c3 [term 3 LEADER]: Becoming Leader. State: Replica: 7fc2639fe7ca454c8de7a2cfce0e63c3, State: Running, Role: LEADER
I20260925 04:42:32.851365 16904 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7fc2639fe7ca454c8de7a2cfce0e63c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fc2639fe7ca454c8de7a2cfce0e63c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38799 } }
I20260925 04:42:32.851646 16900 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7fc2639fe7ca454c8de7a2cfce0e63c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:42:32.852960 16906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fc2639fe7ca454c8de7a2cfce0e63c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7fc2639fe7ca454c8de7a2cfce0e63c3. Latest consensus state: current_term: 3 leader_uuid: "7fc2639fe7ca454c8de7a2cfce0e63c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fc2639fe7ca454c8de7a2cfce0e63c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38799 } } }
I20260925 04:42:32.853091 16906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fc2639fe7ca454c8de7a2cfce0e63c3 [sys.catalog]: This master's current role is: LEADER
I20260925 04:42:32.853384 16905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fc2639fe7ca454c8de7a2cfce0e63c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "7fc2639fe7ca454c8de7a2cfce0e63c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fc2639fe7ca454c8de7a2cfce0e63c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38799 } } }
I20260925 04:42:32.853458 16905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fc2639fe7ca454c8de7a2cfce0e63c3 [sys.catalog]: This master's current role is: LEADER
I20260925 04:42:32.853448 16916 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:42:32.858289 16916 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:42:32.858850 16916 catalog_manager.cc:1259] Loaded cluster ID: 04ecd207d1c04912833509ed3e6cccae
I20260925 04:42:32.858904 16916 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:42:32.860770 16916 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:42:32.861402 16916 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7fc2639fe7ca454c8de7a2cfce0e63c3: Loaded TSK: 0
I20260925 04:42:32.861471 16916 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7fc2639fe7ca454c8de7a2cfce0e63c3: Loaded TSK: 1
F20260925 04:42:32.862087 16916 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f12a8ff39ad google::LogMessage::Fail() at ??:0
@ 0x7f12a8ff688a google::LogMessage::SendToLog() at ??:0
@ 0x7f12a8ff34de google::LogMessage::Flush() at ??:0
@ 0x7f12a8ff4a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f12abcb718d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f12abcaff0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f12b39587bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f12b393299c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f12b3933178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f12b3974df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f12b399382b std::function<>::operator()() at ??:0
@ 0x7f12b3932b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f12b3933f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f12b3930579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f12b3974640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f12ab2453d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f12ab245c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f12ab2473f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f12ab236512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f12ac87a6db start_thread at ??:0
@ 0x7f12a7af161f clone at ??:0
W20260925 04:42:32.958844 16902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:42:32.959115 16902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:42:32.959162 16902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:42:32.963425 16902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:42:32.963506 16902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:42:32.963569 16902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 04:42:32.971374 16902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41203
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35771
--tserver_master_addrs=127.0.98.62:38799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:42:32.972604 16902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:42:32.973654 16902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:42:32.983672 16950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:42:32.983665 16947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:42:32.983665 16948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:42:32.984166 16902 server_base.cc:1034] running on GCE node
I20260925 04:42:32.984596 16902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:42:32.985329 16902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:42:32.986534 16902 hybrid_clock.cc:648] HybridClock initialized: now 1790311352986495 us; error 50 us; skew 500 ppm
Sep 25 04:42:32 dist-test-slave-dc95 krb5kdc[15756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311352, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:42:32.989691 16902 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 04:42:32.991021 16902 webserver.cc:460] Webserver started at http://127.0.98.1:35771/ using document root <none> and password file <none>
I20260925 04:42:32.991828 16902 fs_manager.cc:359] Metadata directory not provided
I20260925 04:42:32.991925 16902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:42:32.997839 16902 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 04:42:33.001096 16957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:33.002843 16902 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:42:33.003011 16902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-0/wal
uuid: "11f96b80805a4844b6650c59f3d524eb"
format_stamp: "Formatted at 2026-09-25 04:41:50 on dist-test-slave-dc95"
server_key: "13b385f8ae8c81800a5364e5d1a66da3"
server_key_iv: "c52952f308cdd5d540d4915cc7bfa80b"
server_key_version: "encryptionkey@0"
I20260925 04:42:33.003855 16902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:42:33.049459 16902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:42:33.053123 16902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:42:33.053323 16902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:42:33.053969 16902 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:42:33.055187 16964 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 04:42:33.061055 16902 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 04:42:33.061161 16902 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260925 04:42:33.061237 16902 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 04:42:33.063758 16902 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 04:42:33.063844 16902 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.003s sys 0.000s
I20260925 04:42:33.064196 16964 tablet_bootstrap.cc:492] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 11f96b80805a4844b6650c59f3d524eb: Bootstrap starting.
I20260925 04:42:33.074185 16902 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41203
I20260925 04:42:33.074496 17071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41203 every 8 connection(s)
I20260925 04:42:33.075287 16902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
I20260925 04:42:33.075740 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 16902
I20260925 04:42:33.076217 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:45283
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:38799
--webserver_port=46187
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:42:33.086830 16964 log.cc:824] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 11f96b80805a4844b6650c59f3d524eb: Log is configured to *not* fsync() on all Append() calls
I20260925 04:42:33.096927 16964 tablet_bootstrap.cc:492] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 11f96b80805a4844b6650c59f3d524eb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 04:42:33.097396 16964 tablet_bootstrap.cc:492] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 11f96b80805a4844b6650c59f3d524eb: Bootstrap complete.
I20260925 04:42:33.098771 16964 ts_tablet_manager.cc:1397] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 11f96b80805a4844b6650c59f3d524eb: Time spent bootstrapping tablet: real 0.035s user 0.023s sys 0.008s
W20260925 04:42:33.100103 17075 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:42:33.078407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38799 (local address 127.0.98.1:37809)
0925 04:42:33.079069 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:33.079136 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0925 04:42:33.080276 (+ 1140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:42:33.099125 (+ 18849us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38799: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":194,"threads_started":1}
W20260925 04:42:33.100683 17072 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38799 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:38799: Client connection negotiation failed: client connection to 127.0.98.62:38799: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 04:42:33.101533 17075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:33.101209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38799 (local address 127.0.98.1:58789)
0925 04:42:33.101404 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:33.101467 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260925 04:42:33.102288 17075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:33.102073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38799 (local address 127.0.98.1:46889)
0925 04:42:33.102186 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:33.102231 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 04:42:33.102504 17072 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 04:42:33.102212 16964 raft_consensus.cc:348] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 11f96b80805a4844b6650c59f3d524eb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b08af7bd450046428e0a48e3edbcb12f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40201 } } peers { permanent_uuid: "699d9f5435bf4e2692984cf3bf4acd8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45283 } } peers { permanent_uuid: "11f96b80805a4844b6650c59f3d524eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41203 } }
I20260925 04:42:33.102794 16964 raft_consensus.cc:729] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 11f96b80805a4844b6650c59f3d524eb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11f96b80805a4844b6650c59f3d524eb, State: Initialized, Role: FOLLOWER
I20260925 04:42:33.103343 16964 consensus_queue.cc:260] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 11f96b80805a4844b6650c59f3d524eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b08af7bd450046428e0a48e3edbcb12f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40201 } } peers { permanent_uuid: "699d9f5435bf4e2692984cf3bf4acd8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45283 } } peers { permanent_uuid: "11f96b80805a4844b6650c59f3d524eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41203 } }
I20260925 04:42:33.104489 16964 ts_tablet_manager.cc:1428] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 11f96b80805a4844b6650c59f3d524eb: Time spent starting tablet: real 0.006s user 0.004s sys 0.004s
I20260925 04:42:33.105042 17075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:33.104629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38799 (local address 127.0.98.1:38921)
0925 04:42:33.104898 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:33.104965 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179}
I20260925 04:42:33.105881 17075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:33.105653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38799 (local address 127.0.98.1:56611)
0925 04:42:33.105772 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:33.105821 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 04:42:33.199815 17074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:42:33.200112 17074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:42:33.200158 17074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:42:33.204416 17074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:42:33.204490 17074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:42:33.204553 17074 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 04:42:33.210278 17074 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:45283
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=46187
--tserver_master_addrs=127.0.98.62:38799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:42:33.211426 17074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:42:33.212497 17074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:42:33.223135 17083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:42:33.223083 17082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:42:33.223083 17085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:42:33.223362 17074 server_base.cc:1034] running on GCE node
I20260925 04:42:33.223909 17074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:42:33.224614 17074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:42:33.225797 17074 hybrid_clock.cc:648] HybridClock initialized: now 1790311353225775 us; error 37 us; skew 500 ppm
Sep 25 04:42:33 dist-test-slave-dc95 krb5kdc[15756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311353, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:42:33.228732 17074 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 04:42:33.229724 17074 webserver.cc:460] Webserver started at http://127.0.98.2:46187/ using document root <none> and password file <none>
I20260925 04:42:33.230301 17074 fs_manager.cc:359] Metadata directory not provided
I20260925 04:42:33.230371 17074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:42:33.234688 17074 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:42:33.237190 17092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:33.238643 17074 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 04:42:33.238788 17074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-1/wal
uuid: "699d9f5435bf4e2692984cf3bf4acd8c"
format_stamp: "Formatted at 2026-09-25 04:41:51 on dist-test-slave-dc95"
server_key: "5b61352dbd08d2fa97c18084f4038d15"
server_key_iv: "ad94de5435669c9804536b871efed19f"
server_key_version: "encryptionkey@0"
I20260925 04:42:33.239413 17074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:42:33.263412 17074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:42:33.266844 17074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:42:33.267084 17074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:42:33.267720 17074 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:42:33.268563 17074 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:42:33.268621 17074 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:33.268675 17074 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:42:33.268698 17074 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:33.277309 17074 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45283
I20260925 04:42:33.277333 17205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45283 every 8 connection(s)
I20260925 04:42:33.278322 17074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
I20260925 04:42:33.282181 17208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:33.280554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38799 (local address 127.0.98.2:43273)
0925 04:42:33.281314 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:33.281436 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":119,"threads_started":1}
W20260925 04:42:33.282673 17206 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38799 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:38799: Client connection negotiation failed: client connection to 127.0.98.62:38799: connect: Connection refused (error 111)
I20260925 04:42:33.283344 17208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:33.283091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38799 (local address 127.0.98.2:57463)
0925 04:42:33.283230 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:33.283283 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260925 04:42:33.284093 17208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:33.283831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38799 (local address 127.0.98.2:51185)
0925 04:42:33.283987 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:33.284037 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 04:42:33.284379 17206 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 04:42:33.285238 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 17074
I20260925 04:42:33.285638 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:40201
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:38799
--webserver_port=41915
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:42:33.400537 17209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:42:33.400827 17209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:42:33.400873 17209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:42:33.405123 17209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:42:33.405211 17209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:42:33.405273 17209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 04:42:33.410831 17209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:40201
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=41915
--tserver_master_addrs=127.0.98.62:38799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:42:33.411999 17209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:42:33.413043 17209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:42:33.423115 17217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:42:33.423116 17214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:42:33.423170 17209 server_base.cc:1034] running on GCE node
W20260925 04:42:33.423116 17215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:42:33.423816 17209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:42:33.424657 17209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:42:33.425832 17209 hybrid_clock.cc:648] HybridClock initialized: now 1790311353425802 us; error 49 us; skew 500 ppm
Sep 25 04:42:33 dist-test-slave-dc95 krb5kdc[15756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311353, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:42:33.428750 17209 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 04:42:33.429735 17209 webserver.cc:460] Webserver started at http://127.0.98.3:41915/ using document root <none> and password file <none>
I20260925 04:42:33.430302 17209 fs_manager.cc:359] Metadata directory not provided
I20260925 04:42:33.430363 17209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:42:33.434597 17209 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:42:33.437045 17224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:33.438486 17209 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 04:42:33.438628 17209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-2/wal
uuid: "b08af7bd450046428e0a48e3edbcb12f"
format_stamp: "Formatted at 2026-09-25 04:41:51 on dist-test-slave-dc95"
server_key: "680b0018850ddae6948b89e79157a822"
server_key_iv: "204bba1af1aca76b5183c55413db3960"
server_key_version: "encryptionkey@0"
I20260925 04:42:33.439219 17209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:42:33.454994 17209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:42:33.458434 17209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:42:33.458624 17209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:42:33.459230 17209 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:42:33.460151 17209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:42:33.460201 17209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:33.460256 17209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:42:33.460280 17209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:33.469086 17209 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40201
I20260925 04:42:33.469120 17337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40201 every 8 connection(s)
I20260925 04:42:33.470134 17209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
I20260925 04:42:33.472391 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 17209
I20260925 04:42:33.474190 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:33.472517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38799 (local address 127.0.98.3:41429)
0925 04:42:33.473301 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:33.473446 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":160,"threads_started":1}
W20260925 04:42:33.474695 17338 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38799 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:38799: Client connection negotiation failed: client connection to 127.0.98.62:38799: connect: Connection refused (error 111)
I20260925 04:42:33.475313 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:33.475095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38799 (local address 127.0.98.3:38009)
0925 04:42:33.475206 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:33.475256 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 04:42:33.475868 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:33.475694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38799 (local address 127.0.98.3:40057)
0925 04:42:33.475780 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:33.475820 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 04:42:33.476018 17338 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 04:42:34.107134 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:34.106679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38799 (local address 127.0.98.1:51899)
0925 04:42:34.106956 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:34.107019 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":76,"threads_started":1}
I20260925 04:42:34.285535 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:34.285067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38799 (local address 127.0.98.2:41941)
0925 04:42:34.285350 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:34.285416 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":84,"threads_started":1}
I20260925 04:42:34.477025 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:34.476608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38799 (local address 127.0.98.3:38939)
0925 04:42:34.476853 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:34.476916 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":71,"threads_started":1}
I20260925 04:42:34.645567 17345 raft_consensus.cc:482] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 11f96b80805a4844b6650c59f3d524eb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:42:34.645773 17345 raft_consensus.cc:504] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 11f96b80805a4844b6650c59f3d524eb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b08af7bd450046428e0a48e3edbcb12f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40201 } } peers { permanent_uuid: "699d9f5435bf4e2692984cf3bf4acd8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45283 } } peers { permanent_uuid: "11f96b80805a4844b6650c59f3d524eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41203 } }
I20260925 04:42:34.646889 17345 leader_election.cc:288] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 11f96b80805a4844b6650c59f3d524eb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b08af7bd450046428e0a48e3edbcb12f (127.0.98.3:40201), 699d9f5435bf4e2692984cf3bf4acd8c (127.0.98.2:45283)
Sep 25 04:42:34 dist-test-slave-dc95 krb5kdc[15756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311352, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 25 04:42:34 dist-test-slave-dc95 krb5kdc[15756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311352, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260925 04:42:34.660816 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:34.647516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45283 (local address 127.0.98.1:43847)
0925 04:42:34.647769 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:34.647780 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 04:42:34.647871 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:42:34.649633 (+ 1762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:42:34.649635 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:42:34.649907 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:42:34.650420 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:34.650435 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:34.651742 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:34.651744 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:34.652709 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:34.652718 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:34.654445 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:34.654449 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:34.654539 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:34.654935 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:42:34.654949 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:42:34.656400 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:42:34.658313 (+ 1913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:42:34.658316 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:42:34.658317 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:42:34.658586 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:42:34.659013 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:42:34.659016 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:42:34.659018 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:42:34.659117 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:42:34.660021 (+ 904us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:42:34.660024 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:42:34.660192 (+ 168us) client_negotiation.cc:769] Sending connection context
0925 04:42:34.660380 (+ 188us) client_negotiation.cc:240] Negotiation successful
0925 04:42:34.660660 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":109,"spinlock_wait_cycles":8192,"thread_start_us":74,"threads_started":1}
I20260925 04:42:34.660949 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:34.647287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56831 (local address 127.0.98.3:40201)
0925 04:42:34.647858 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:34.647883 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:34.647928 (+ 45us) server_negotiation.cc:407] Connection header received
0925 04:42:34.648641 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:34.648647 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:34.648997 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:34.649257 (+ 260us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:42:34.650599 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:34.651564 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:34.652832 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:34.653341 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:34.653499 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:34.656311 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:42:34.656319 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:42:34.656338 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:42:34.656378 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:42:34.658059 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:42:34.658673 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:42:34.658680 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:42:34.658686 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:42:34.658814 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:42:34.659207 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:42:34.659210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:42:34.659211 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:42:34.659810 (+ 599us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:42:34.659950 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:34.660509 (+ 559us) server_negotiation.cc:299] Negotiation successful
0925 04:42:34.660772 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":97,"threads_started":1}
I20260925 04:42:34.660949 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:34.647159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40201 (local address 127.0.98.1:56831)
0925 04:42:34.647436 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:34.647453 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:42:34.647575 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:42:34.649365 (+ 1790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:42:34.649371 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:42:34.649910 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:42:34.650420 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:34.650436 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:34.651733 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:34.651736 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:34.652703 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:34.652713 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:34.653466 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:34.653469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:34.653647 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:34.654310 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:42:34.654330 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:42:34.656033 (+ 1703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:42:34.658184 (+ 2151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:42:34.658190 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:42:34.658203 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:42:34.658552 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:42:34.658917 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:42:34.658920 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:42:34.658922 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:42:34.659102 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:42:34.659937 (+ 835us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:42:34.659944 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:42:34.660194 (+ 250us) client_negotiation.cc:769] Sending connection context
0925 04:42:34.660381 (+ 187us) client_negotiation.cc:240] Negotiation successful
0925 04:42:34.660753 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":179,"spinlock_wait_cycles":53632,"thread_start_us":66,"threads_started":1}
I20260925 04:42:34.661715 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:34.647710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43847 (local address 127.0.98.2:45283)
0925 04:42:34.648025 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:34.648043 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:34.648072 (+ 29us) server_negotiation.cc:407] Connection header received
0925 04:42:34.648687 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:34.648692 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:34.648998 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:34.649266 (+ 268us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:42:34.650587 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:34.651564 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:34.653722 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:34.654311 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:34.654468 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:34.656578 (+ 2110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:42:34.656583 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:42:34.656593 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:42:34.656622 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:42:34.658189 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:42:34.658699 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:42:34.658704 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:42:34.658707 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:42:34.658811 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:42:34.659231 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:42:34.659235 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:42:34.659236 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:42:34.659850 (+ 614us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:42:34.659975 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:34.661260 (+ 1285us) server_negotiation.cc:299] Negotiation successful
0925 04:42:34.661547 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":127,"threads_started":1}
W20260925 04:42:34.663585 16961 leader_election.cc:341] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 11f96b80805a4844b6650c59f3d524eb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b08af7bd450046428e0a48e3edbcb12f (127.0.98.3:40201): Not found: Tablet not found: 314ff12e791e4ceb88d0bc5a81d3c6f9
W20260925 04:42:34.663837 16961 leader_election.cc:341] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 11f96b80805a4844b6650c59f3d524eb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 699d9f5435bf4e2692984cf3bf4acd8c (127.0.98.2:45283): Not found: Tablet not found: 314ff12e791e4ceb88d0bc5a81d3c6f9
I20260925 04:42:34.663918 16961 leader_election.cc:302] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 11f96b80805a4844b6650c59f3d524eb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 11f96b80805a4844b6650c59f3d524eb; no voters: 699d9f5435bf4e2692984cf3bf4acd8c, b08af7bd450046428e0a48e3edbcb12f
I20260925 04:42:34.664165 17345 raft_consensus.cc:2726] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 11f96b80805a4844b6650c59f3d524eb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 04:42:35.108235 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:35.107922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38799 (local address 127.0.98.1:33543)
0925 04:42:35.108063 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:35.108153 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260925 04:42:35.286751 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:35.286338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38799 (local address 127.0.98.2:39331)
0925 04:42:35.286591 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:35.286653 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":75,"threads_started":1}
I20260925 04:42:35.478214 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:35.477787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38799 (local address 127.0.98.3:47497)
0925 04:42:35.478052 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:35.478114 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":74,"threads_started":1}
I20260925 04:42:36.109576 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:36.109059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38799 (local address 127.0.98.1:56227)
0925 04:42:36.109418 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:36.109479 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":92,"threads_started":1}
I20260925 04:42:36.288146 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:36.287685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38799 (local address 127.0.98.2:49847)
0925 04:42:36.287949 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:36.288017 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":74,"threads_started":1}
I20260925 04:42:36.303263 17354 raft_consensus.cc:482] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 11f96b80805a4844b6650c59f3d524eb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:42:36.303435 17354 raft_consensus.cc:504] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 11f96b80805a4844b6650c59f3d524eb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b08af7bd450046428e0a48e3edbcb12f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40201 } } peers { permanent_uuid: "699d9f5435bf4e2692984cf3bf4acd8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45283 } } peers { permanent_uuid: "11f96b80805a4844b6650c59f3d524eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41203 } }
I20260925 04:42:36.303879 17354 leader_election.cc:288] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 11f96b80805a4844b6650c59f3d524eb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b08af7bd450046428e0a48e3edbcb12f (127.0.98.3:40201), 699d9f5435bf4e2692984cf3bf4acd8c (127.0.98.2:45283)
W20260925 04:42:36.304718 16961 leader_election.cc:341] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 11f96b80805a4844b6650c59f3d524eb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b08af7bd450046428e0a48e3edbcb12f (127.0.98.3:40201): Not found: Tablet not found: 314ff12e791e4ceb88d0bc5a81d3c6f9
W20260925 04:42:36.304841 16961 leader_election.cc:341] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 11f96b80805a4844b6650c59f3d524eb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 699d9f5435bf4e2692984cf3bf4acd8c (127.0.98.2:45283): Not found: Tablet not found: 314ff12e791e4ceb88d0bc5a81d3c6f9
I20260925 04:42:36.304879 16961 leader_election.cc:302] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 11f96b80805a4844b6650c59f3d524eb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 11f96b80805a4844b6650c59f3d524eb; no voters: 699d9f5435bf4e2692984cf3bf4acd8c, b08af7bd450046428e0a48e3edbcb12f
I20260925 04:42:36.305027 17354 raft_consensus.cc:2726] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 11f96b80805a4844b6650c59f3d524eb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 04:42:36.479552 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:36.479028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38799 (local address 127.0.98.3:59201)
0925 04:42:36.479314 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:36.479432 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":90,"threads_started":1}
I20260925 04:42:37.110659 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:37.110255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38799 (local address 127.0.98.1:60971)
0925 04:42:37.110502 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:37.110561 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":74,"threads_started":1}
I20260925 04:42:37.289492 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:37.289011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38799 (local address 127.0.98.2:45237)
0925 04:42:37.289304 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:37.289366 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":93,"threads_started":1}
I20260925 04:42:37.481032 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:37.480505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38799 (local address 127.0.98.3:40349)
0925 04:42:37.480806 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:37.480886 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":87,"threads_started":1}
I20260925 04:42:38.112084 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:38.111660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38799 (local address 127.0.98.1:60303)
0925 04:42:38.111924 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:38.111986 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":90,"threads_started":1}
I20260925 04:42:38.290943 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:38.290448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38799 (local address 127.0.98.2:46511)
0925 04:42:38.290765 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:38.290831 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":95,"threads_started":1}
I20260925 04:42:38.482550 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:38.482003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38799 (local address 127.0.98.3:51279)
0925 04:42:38.482326 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:38.482388 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":83,"threads_started":1}
I20260925 04:42:39.113369 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:39.112915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38799 (local address 127.0.98.1:51341)
0925 04:42:39.113210 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:39.113272 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":75,"threads_started":1}
I20260925 04:42:39.118507 17363 raft_consensus.cc:482] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 11f96b80805a4844b6650c59f3d524eb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:42:39.118647 17363 raft_consensus.cc:504] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 11f96b80805a4844b6650c59f3d524eb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b08af7bd450046428e0a48e3edbcb12f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40201 } } peers { permanent_uuid: "699d9f5435bf4e2692984cf3bf4acd8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45283 } } peers { permanent_uuid: "11f96b80805a4844b6650c59f3d524eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41203 } }
I20260925 04:42:39.119067 17363 leader_election.cc:288] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 11f96b80805a4844b6650c59f3d524eb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b08af7bd450046428e0a48e3edbcb12f (127.0.98.3:40201), 699d9f5435bf4e2692984cf3bf4acd8c (127.0.98.2:45283)
W20260925 04:42:39.119906 16961 leader_election.cc:341] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 11f96b80805a4844b6650c59f3d524eb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b08af7bd450046428e0a48e3edbcb12f (127.0.98.3:40201): Not found: Tablet not found: 314ff12e791e4ceb88d0bc5a81d3c6f9
W20260925 04:42:39.120069 16961 leader_election.cc:341] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 11f96b80805a4844b6650c59f3d524eb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 699d9f5435bf4e2692984cf3bf4acd8c (127.0.98.2:45283): Not found: Tablet not found: 314ff12e791e4ceb88d0bc5a81d3c6f9
I20260925 04:42:39.120137 16961 leader_election.cc:302] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 11f96b80805a4844b6650c59f3d524eb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 11f96b80805a4844b6650c59f3d524eb; no voters: 699d9f5435bf4e2692984cf3bf4acd8c, b08af7bd450046428e0a48e3edbcb12f
I20260925 04:42:39.120289 17363 raft_consensus.cc:2726] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 11f96b80805a4844b6650c59f3d524eb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 04:42:39.292402 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:39.291896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38799 (local address 127.0.98.2:40105)
0925 04:42:39.292229 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:39.292298 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":92,"threads_started":1}
I20260925 04:42:39.484005 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:39.483455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38799 (local address 127.0.98.3:49491)
0925 04:42:39.483797 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:39.483861 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":79,"threads_started":1}
I20260925 04:42:40.114835 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:40.114296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38799 (local address 127.0.98.1:43007)
0925 04:42:40.114630 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:40.114696 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":96,"threads_started":1}
I20260925 04:42:40.293798 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:40.293230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38799 (local address 127.0.98.2:58253)
0925 04:42:40.293631 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:40.293695 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":85,"threads_started":1}
I20260925 04:42:40.485507 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:40.484943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38799 (local address 127.0.98.3:33251)
0925 04:42:40.485290 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:40.485354 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":85,"threads_started":1}
I20260925 04:42:41.116313 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:41.115873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38799 (local address 127.0.98.1:50957)
0925 04:42:41.116146 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:41.116210 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":84,"threads_started":1}
I20260925 04:42:41.295267 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:41.294717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38799 (local address 127.0.98.2:47611)
0925 04:42:41.295040 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:41.295153 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":92,"threads_started":1}
I20260925 04:42:41.486788 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:41.486353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38799 (local address 127.0.98.3:58957)
0925 04:42:41.486634 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:41.486692 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":81,"threads_started":1}
I20260925 04:42:42.117506 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:42.117087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38799 (local address 127.0.98.1:33397)
0925 04:42:42.117347 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:42.117408 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":78,"threads_started":1}
I20260925 04:42:42.296617 17373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:42.296163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38799 (local address 127.0.98.2:50889)
0925 04:42:42.296443 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:42.296510 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":81,"threads_started":1}
I20260925 04:42:42.342700 17374 raft_consensus.cc:482] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 11f96b80805a4844b6650c59f3d524eb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:42:42.342944 17374 raft_consensus.cc:504] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 11f96b80805a4844b6650c59f3d524eb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b08af7bd450046428e0a48e3edbcb12f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40201 } } peers { permanent_uuid: "699d9f5435bf4e2692984cf3bf4acd8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45283 } } peers { permanent_uuid: "11f96b80805a4844b6650c59f3d524eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41203 } }
I20260925 04:42:42.343384 17374 leader_election.cc:288] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 11f96b80805a4844b6650c59f3d524eb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b08af7bd450046428e0a48e3edbcb12f (127.0.98.3:40201), 699d9f5435bf4e2692984cf3bf4acd8c (127.0.98.2:45283)
W20260925 04:42:42.344271 16961 leader_election.cc:341] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 11f96b80805a4844b6650c59f3d524eb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b08af7bd450046428e0a48e3edbcb12f (127.0.98.3:40201): Not found: Tablet not found: 314ff12e791e4ceb88d0bc5a81d3c6f9
W20260925 04:42:42.344406 16961 leader_election.cc:341] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 11f96b80805a4844b6650c59f3d524eb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 699d9f5435bf4e2692984cf3bf4acd8c (127.0.98.2:45283): Not found: Tablet not found: 314ff12e791e4ceb88d0bc5a81d3c6f9
I20260925 04:42:42.344460 16961 leader_election.cc:302] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 11f96b80805a4844b6650c59f3d524eb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 11f96b80805a4844b6650c59f3d524eb; no voters: 699d9f5435bf4e2692984cf3bf4acd8c, b08af7bd450046428e0a48e3edbcb12f
I20260925 04:42:42.344601 17374 raft_consensus.cc:2726] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 11f96b80805a4844b6650c59f3d524eb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 04:42:42.488200 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:42.487770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38799 (local address 127.0.98.3:54241)
0925 04:42:42.488034 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:42.488100 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":77,"threads_started":1}
I20260925 04:42:43.118969 17376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:43.118515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38799 (local address 127.0.98.1:36143)
0925 04:42:43.118804 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:43.118869 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":108,"threads_started":1}
I20260925 04:42:43.298095 17377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:43.297608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38799 (local address 127.0.98.2:51599)
0925 04:42:43.297899 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:43.297958 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":82,"threads_started":1}
I20260925 04:42:43.489494 17378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:43.489018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38799 (local address 127.0.98.3:33161)
0925 04:42:43.489328 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:43.489392 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":91,"threads_started":1}
I20260925 04:42:44.120323 17379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:44.119848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38799 (local address 127.0.98.1:49887)
0925 04:42:44.120152 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:44.120217 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":90,"threads_started":1}
I20260925 04:42:44.299427 17380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:44.298947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38799 (local address 127.0.98.2:42683)
0925 04:42:44.299261 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:44.299326 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":76,"threads_started":1}
I20260925 04:42:44.490834 17381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:44.490401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38799 (local address 127.0.98.3:36585)
0925 04:42:44.490677 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:44.490737 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":91,"threads_started":1}
I20260925 04:42:45.121803 17382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:45.121303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38799 (local address 127.0.98.1:43737)
0925 04:42:45.121608 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:45.121672 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":96,"threads_started":1}
I20260925 04:42:45.300889 17383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:45.300385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38799 (local address 127.0.98.2:43789)
0925 04:42:45.300721 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:45.300786 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":82,"threads_started":1}
I20260925 04:42:45.492264 17384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:45.491788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38799 (local address 127.0.98.3:49733)
0925 04:42:45.492093 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:45.492166 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":89,"threads_started":1}
I20260925 04:42:45.826018 17385 raft_consensus.cc:482] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 11f96b80805a4844b6650c59f3d524eb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:42:45.826192 17385 raft_consensus.cc:504] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 11f96b80805a4844b6650c59f3d524eb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b08af7bd450046428e0a48e3edbcb12f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40201 } } peers { permanent_uuid: "699d9f5435bf4e2692984cf3bf4acd8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45283 } } peers { permanent_uuid: "11f96b80805a4844b6650c59f3d524eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41203 } }
I20260925 04:42:45.826625 17385 leader_election.cc:288] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 11f96b80805a4844b6650c59f3d524eb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b08af7bd450046428e0a48e3edbcb12f (127.0.98.3:40201), 699d9f5435bf4e2692984cf3bf4acd8c (127.0.98.2:45283)
W20260925 04:42:45.827368 16961 leader_election.cc:341] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 11f96b80805a4844b6650c59f3d524eb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b08af7bd450046428e0a48e3edbcb12f (127.0.98.3:40201): Not found: Tablet not found: 314ff12e791e4ceb88d0bc5a81d3c6f9
W20260925 04:42:45.827492 16961 leader_election.cc:341] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 11f96b80805a4844b6650c59f3d524eb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 699d9f5435bf4e2692984cf3bf4acd8c (127.0.98.2:45283): Not found: Tablet not found: 314ff12e791e4ceb88d0bc5a81d3c6f9
I20260925 04:42:45.827529 16961 leader_election.cc:302] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 11f96b80805a4844b6650c59f3d524eb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 11f96b80805a4844b6650c59f3d524eb; no voters: 699d9f5435bf4e2692984cf3bf4acd8c, b08af7bd450046428e0a48e3edbcb12f
I20260925 04:42:45.827692 17385 raft_consensus.cc:2726] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 11f96b80805a4844b6650c59f3d524eb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 04:42:46.123082 17386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:46.122672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38799 (local address 127.0.98.1:46201)
0925 04:42:46.122921 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:46.122983 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":77,"threads_started":1}
I20260925 04:42:46.302227 17387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:46.301793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38799 (local address 127.0.98.2:34139)
0925 04:42:46.302072 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:46.302129 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":77,"threads_started":1}
I20260925 04:42:46.493574 17388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:46.493140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38799 (local address 127.0.98.3:33727)
0925 04:42:46.493417 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:46.493478 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":80,"threads_started":1}
I20260925 04:42:47.124480 17389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:47.124039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38799 (local address 127.0.98.1:47077)
0925 04:42:47.124314 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:47.124383 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":74,"threads_started":1}
I20260925 04:42:47.303761 17390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:47.303219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38799 (local address 127.0.98.2:32999)
0925 04:42:47.303530 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:47.303598 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":85,"threads_started":1}
I20260925 04:42:47.494769 17391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:47.494353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38799 (local address 127.0.98.3:35141)
0925 04:42:47.494612 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:47.494670 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":75,"threads_started":1}
I20260925 04:42:47.581032 17392 raft_consensus.cc:482] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 11f96b80805a4844b6650c59f3d524eb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:42:47.581173 17392 raft_consensus.cc:504] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 11f96b80805a4844b6650c59f3d524eb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b08af7bd450046428e0a48e3edbcb12f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40201 } } peers { permanent_uuid: "699d9f5435bf4e2692984cf3bf4acd8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45283 } } peers { permanent_uuid: "11f96b80805a4844b6650c59f3d524eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41203 } }
I20260925 04:42:47.581602 17392 leader_election.cc:288] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 11f96b80805a4844b6650c59f3d524eb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b08af7bd450046428e0a48e3edbcb12f (127.0.98.3:40201), 699d9f5435bf4e2692984cf3bf4acd8c (127.0.98.2:45283)
W20260925 04:42:47.582415 16961 leader_election.cc:341] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 11f96b80805a4844b6650c59f3d524eb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b08af7bd450046428e0a48e3edbcb12f (127.0.98.3:40201): Not found: Tablet not found: 314ff12e791e4ceb88d0bc5a81d3c6f9
W20260925 04:42:47.582517 16961 leader_election.cc:341] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 11f96b80805a4844b6650c59f3d524eb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 699d9f5435bf4e2692984cf3bf4acd8c (127.0.98.2:45283): Not found: Tablet not found: 314ff12e791e4ceb88d0bc5a81d3c6f9
I20260925 04:42:47.582554 16961 leader_election.cc:302] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 11f96b80805a4844b6650c59f3d524eb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 11f96b80805a4844b6650c59f3d524eb; no voters: 699d9f5435bf4e2692984cf3bf4acd8c, b08af7bd450046428e0a48e3edbcb12f
I20260925 04:42:47.582715 17392 raft_consensus.cc:2726] T 314ff12e791e4ceb88d0bc5a81d3c6f9 P 11f96b80805a4844b6650c59f3d524eb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 04:42:48.125869 17393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:48.125447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38799 (local address 127.0.98.1:59751)
0925 04:42:48.125715 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:48.125775 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":82,"threads_started":1}
I20260925 04:42:48.305236 17394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:48.304781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38799 (local address 127.0.98.2:47469)
0925 04:42:48.305064 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:48.305129 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":87,"threads_started":1}
I20260925 04:42:48.476470 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 16902
I20260925 04:42:48.481977 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 17074
I20260925 04:42:48.487085 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 17209
2026-09-25T04:42:48Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59199 ms)
[----------] 30 tests from SecurityITest (244998 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:42:48 dist-test-slave-dc95 krb5kdc[17397](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:42:48 dist-test-slave-dc95 krb5kdc[17397](info): set up 2 sockets
Sep 25 04:42:48 dist-test-slave-dc95 krb5kdc[17397](info): commencing operation
krb5kdc: starting...
W20260925 04:42:49.463357 392 mini_kdc.cc:121] Time spent starting KDC: real 0.955s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/krb5kdc/test-user.keytab.
Sep 25 04:42:49 dist-test-slave-dc95 krb5kdc[17397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:42:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:42:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 04:42:49.588207 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37141
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37141
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:42:49.699846 17410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:42:49.700140 17410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:42:49.700182 17410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:42:49.704404 17410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:42:49.704483 17410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:42:49.704501 17410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:42:49.704519 17410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:42:49.704532 17410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:42:49.710160 17410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:42:49.711340 17410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:42:49.712425 17410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:42:49.721431 17416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:42:49.721529 17418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:42:49.721519 17415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:42:49.721496 17410 server_base.cc:1034] running on GCE node
I20260925 04:42:49.722195 17410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:42:49.723342 17410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:42:49.724539 17410 hybrid_clock.cc:648] HybridClock initialized: now 1790311369724521 us; error 31 us; skew 500 ppm
Sep 25 04:42:49 dist-test-slave-dc95 krb5kdc[17397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311369, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:42:49.727813 17410 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 04:42:49.728818 17410 webserver.cc:460] Webserver started at http://127.0.98.62:36769/ using document root <none> and password file <none>
I20260925 04:42:49.729362 17410 fs_manager.cc:359] Metadata directory not provided
I20260925 04:42:49.729420 17410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:42:49.729588 17410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:42:49.731671 17410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/minicluster-data/master-0/data/instance:
uuid: "eab965905936434a915e5d3fc010dff7"
format_stamp: "Formatted at 2026-09-25 04:42:49 on dist-test-slave-dc95"
server_key: "d4e990d665b03de7888c90939431f52e"
server_key_iv: "2ed4a9b3ae7b8d675ff2c029c7f178a8"
server_key_version: "encryptionkey@0"
I20260925 04:42:49.732226 17410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/minicluster-data/master-0/wal/instance:
uuid: "eab965905936434a915e5d3fc010dff7"
format_stamp: "Formatted at 2026-09-25 04:42:49 on dist-test-slave-dc95"
server_key: "d4e990d665b03de7888c90939431f52e"
server_key_iv: "2ed4a9b3ae7b8d675ff2c029c7f178a8"
server_key_version: "encryptionkey@0"
I20260925 04:42:49.736061 17410 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 04:42:49.738560 17425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:49.739872 17410 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:42:49.740036 17410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/minicluster-data/master-0/wal
uuid: "eab965905936434a915e5d3fc010dff7"
format_stamp: "Formatted at 2026-09-25 04:42:49 on dist-test-slave-dc95"
server_key: "d4e990d665b03de7888c90939431f52e"
server_key_iv: "2ed4a9b3ae7b8d675ff2c029c7f178a8"
server_key_version: "encryptionkey@0"
I20260925 04:42:49.740144 17410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:42:49.757531 17410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:42:49.761021 17410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:42:49.761200 17410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:42:49.768456 17410 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37141
I20260925 04:42:49.768497 17477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37141 every 8 connection(s)
I20260925 04:42:49.769447 17410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
I20260925 04:42:49.772409 17478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:42:49.774677 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 17410
I20260925 04:42:49.774839 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/minicluster-data/master-0/wal/instance
I20260925 04:42:49.775147 392 external_mini_cluster.cc:1428] Setting key fec3bafc4f9a17cda2a6bab9be1bdf04
I20260925 04:42:49.779026 17478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eab965905936434a915e5d3fc010dff7: Bootstrap starting.
I20260925 04:42:49.781347 17478 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eab965905936434a915e5d3fc010dff7: Neither blocks nor log segments found. Creating new log.
I20260925 04:42:49.782284 17478 log.cc:824] T 00000000000000000000000000000000 P eab965905936434a915e5d3fc010dff7: Log is configured to *not* fsync() on all Append() calls
Sep 25 04:42:49 dist-test-slave-dc95 krb5kdc[17397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:42:49.784585 17478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eab965905936434a915e5d3fc010dff7: No bootstrap required, opened a new log
I20260925 04:42:49.788684 17478 raft_consensus.cc:348] T 00000000000000000000000000000000 P eab965905936434a915e5d3fc010dff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eab965905936434a915e5d3fc010dff7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37141 } }
I20260925 04:42:49.788944 17478 raft_consensus.cc:374] T 00000000000000000000000000000000 P eab965905936434a915e5d3fc010dff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:42:49.788980 17478 raft_consensus.cc:729] T 00000000000000000000000000000000 P eab965905936434a915e5d3fc010dff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eab965905936434a915e5d3fc010dff7, State: Initialized, Role: FOLLOWER
I20260925 04:42:49.789475 17478 consensus_queue.cc:260] T 00000000000000000000000000000000 P eab965905936434a915e5d3fc010dff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eab965905936434a915e5d3fc010dff7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37141 } }
I20260925 04:42:49.789623 17478 raft_consensus.cc:388] T 00000000000000000000000000000000 P eab965905936434a915e5d3fc010dff7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:42:49.789705 17478 raft_consensus.cc:482] T 00000000000000000000000000000000 P eab965905936434a915e5d3fc010dff7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:42:49.789829 17478 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eab965905936434a915e5d3fc010dff7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:42:49.790762 17481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:49.776705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59734 (local address 127.0.98.62:37141)
0925 04:42:49.777230 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:49.777247 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:49.777288 (+ 41us) server_negotiation.cc:407] Connection header received
0925 04:42:49.778087 (+ 799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:49.778107 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:49.778490 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:49.778848 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:42:49.779689 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:49.780697 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:49.781606 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:49.782320 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:49.782488 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:49.785018 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:42:49.785025 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:42:49.785044 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:42:49.785079 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:42:49.787657 (+ 2578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:42:49.788157 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:42:49.788163 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:42:49.788169 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:42:49.788299 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:42:49.788587 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:42:49.788591 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:42:49.788593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:42:49.789069 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:42:49.789280 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:49.789553 (+ 273us) server_negotiation.cc:299] Negotiation successful
0925 04:42:49.789887 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":159,"threads_started":1}
I20260925 04:42:49.790808 17478 raft_consensus.cc:504] T 00000000000000000000000000000000 P eab965905936434a915e5d3fc010dff7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eab965905936434a915e5d3fc010dff7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37141 } }
I20260925 04:42:49.791173 17478 leader_election.cc:302] T 00000000000000000000000000000000 P eab965905936434a915e5d3fc010dff7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eab965905936434a915e5d3fc010dff7; no voters:
I20260925 04:42:49.791487 17478 leader_election.cc:288] T 00000000000000000000000000000000 P eab965905936434a915e5d3fc010dff7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:42:49.791579 17483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eab965905936434a915e5d3fc010dff7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:42:49.791831 17483 raft_consensus.cc:686] T 00000000000000000000000000000000 P eab965905936434a915e5d3fc010dff7 [term 1 LEADER]: Becoming Leader. State: Replica: eab965905936434a915e5d3fc010dff7, State: Running, Role: LEADER
I20260925 04:42:49.792209 17483 consensus_queue.cc:237] T 00000000000000000000000000000000 P eab965905936434a915e5d3fc010dff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eab965905936434a915e5d3fc010dff7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37141 } }
I20260925 04:42:49.792905 17478 sys_catalog.cc:564] T 00000000000000000000000000000000 P eab965905936434a915e5d3fc010dff7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:42:49.793529 17485 sys_catalog.cc:455] T 00000000000000000000000000000000 P eab965905936434a915e5d3fc010dff7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eab965905936434a915e5d3fc010dff7. Latest consensus state: current_term: 1 leader_uuid: "eab965905936434a915e5d3fc010dff7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eab965905936434a915e5d3fc010dff7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37141 } } }
I20260925 04:42:49.793661 17485 sys_catalog.cc:458] T 00000000000000000000000000000000 P eab965905936434a915e5d3fc010dff7 [sys.catalog]: This master's current role is: LEADER
I20260925 04:42:49.793872 17484 sys_catalog.cc:455] T 00000000000000000000000000000000 P eab965905936434a915e5d3fc010dff7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eab965905936434a915e5d3fc010dff7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eab965905936434a915e5d3fc010dff7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37141 } } }
I20260925 04:42:49.794061 17484 sys_catalog.cc:458] T 00000000000000000000000000000000 P eab965905936434a915e5d3fc010dff7 [sys.catalog]: This master's current role is: LEADER
I20260925 04:42:49.794165 17489 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:42:49.797317 17489 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:42:49.803471 17489 catalog_manager.cc:1347] Generated new cluster ID: 13d7aab3f67141df92e3d5159b63496a
I20260925 04:42:49.803562 17489 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:42:49.822516 17489 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:42:49.823948 17489 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:42:49.835116 17489 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eab965905936434a915e5d3fc010dff7: Generated new TSK 0
I20260925 04:42:49.836094 17489 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 04:42:49.879487 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37141
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:42:49.995225 17504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:42:49.995527 17504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:42:49.995591 17504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:42:49.999923 17504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:42:50.000000 17504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:42:50.000061 17504 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 04:42:50.006795 17504 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:42:50.008051 17504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:42:50.009260 17504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:42:50.019187 17510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:42:50.019201 17512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:42:50.019215 17509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:42:50.019424 17504 server_base.cc:1034] running on GCE node
I20260925 04:42:50.019949 17504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:42:50.020630 17504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:42:50.021822 17504 hybrid_clock.cc:648] HybridClock initialized: now 1790311370021799 us; error 37 us; skew 500 ppm
Sep 25 04:42:50 dist-test-slave-dc95 krb5kdc[17397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311370, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:42:50.025055 17504 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 04:42:50.026077 17504 webserver.cc:460] Webserver started at http://127.0.98.1:35375/ using document root <none> and password file <none>
I20260925 04:42:50.026640 17504 fs_manager.cc:359] Metadata directory not provided
I20260925 04:42:50.026705 17504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:42:50.026897 17504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:42:50.029114 17504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/minicluster-data/ts-0/data/instance:
uuid: "6d166e6940f844a1a6aadc712c6bb596"
format_stamp: "Formatted at 2026-09-25 04:42:50 on dist-test-slave-dc95"
server_key: "ddc7c050254b7c95a7824484e2780a80"
server_key_iv: "92412a13254f0f8392223e3e98438f30"
server_key_version: "encryptionkey@0"
I20260925 04:42:50.029711 17504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6d166e6940f844a1a6aadc712c6bb596"
format_stamp: "Formatted at 2026-09-25 04:42:50 on dist-test-slave-dc95"
server_key: "ddc7c050254b7c95a7824484e2780a80"
server_key_iv: "92412a13254f0f8392223e3e98438f30"
server_key_version: "encryptionkey@0"
I20260925 04:42:50.033741 17504 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 04:42:50.036842 17519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:50.038291 17504 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:42:50.038426 17504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/minicluster-data/ts-0/wal
uuid: "6d166e6940f844a1a6aadc712c6bb596"
format_stamp: "Formatted at 2026-09-25 04:42:50 on dist-test-slave-dc95"
server_key: "ddc7c050254b7c95a7824484e2780a80"
server_key_iv: "92412a13254f0f8392223e3e98438f30"
server_key_version: "encryptionkey@0"
I20260925 04:42:50.038546 17504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:42:50.050797 17504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:42:50.054579 17504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:42:50.054809 17504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:42:50.055415 17504 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:42:50.056432 17504 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:42:50.056480 17504 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:50.056536 17504 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:42:50.056564 17504 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:50.066877 17504 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35915
I20260925 04:42:50.066913 17632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35915 every 8 connection(s)
I20260925 04:42:50.067914 17504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
I20260925 04:42:50.076766 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 17504
I20260925 04:42:50.076910 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/minicluster-data/ts-0/wal/instance
I20260925 04:42:50.077201 392 external_mini_cluster.cc:1428] Setting key f7edea7a0f6156bf8da86eaec85220aa
Sep 25 04:42:50 dist-test-slave-dc95 krb5kdc[17397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311370, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:42:50.082314 17481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:50.069900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52505 (local address 127.0.98.62:37141)
0925 04:42:50.070034 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:50.070038 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:50.070868 (+ 830us) server_negotiation.cc:407] Connection header received
0925 04:42:50.071828 (+ 960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:50.071831 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:50.071891 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:50.071985 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:42:50.073538 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:50.074090 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:50.075348 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:50.075588 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:50.075707 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:50.078550 (+ 2843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:42:50.078554 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:42:50.078557 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:42:50.078587 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:42:50.080213 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:42:50.080737 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:42:50.080741 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:42:50.080743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:42:50.080839 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:42:50.081154 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:42:50.081157 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:42:50.081159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:42:50.081388 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:42:50.081502 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:50.082048 (+ 546us) server_negotiation.cc:299] Negotiation successful
0925 04:42:50.082186 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 04:42:50.083514 17635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:50.070212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37141 (local address 127.0.98.1:52505)
0925 04:42:50.070703 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:50.070750 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 04:42:50.071578 (+ 828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:42:50.072233 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:42:50.072240 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:42:50.072796 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:42:50.073352 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:50.073369 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:50.074237 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:50.074240 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:50.075217 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:50.075228 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:50.075822 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:50.075826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:50.076001 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:50.076603 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:42:50.076621 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:42:50.078349 (+ 1728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:42:50.080338 (+ 1989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:42:50.080344 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:42:50.080360 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:42:50.080634 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:42:50.080926 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:42:50.080929 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:42:50.080930 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:42:50.081073 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:42:50.081490 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:42:50.081495 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:42:50.081794 (+ 299us) client_negotiation.cc:769] Sending connection context
0925 04:42:50.082002 (+ 208us) client_negotiation.cc:240] Negotiation successful
0925 04:42:50.082389 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":126,"threads_started":1}
I20260925 04:42:50.084910 17633 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37141
I20260925 04:42:50.085153 17633 heartbeater.cc:464] Registering TS with master...
I20260925 04:42:50.085726 17633 heartbeater.cc:511] Master 127.0.98.62:37141 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 04:42:50.087494 17442 ts_manager.cc:194] Registered new tserver with Master: 6d166e6940f844a1a6aadc712c6bb596 (127.0.98.1:35915)
I20260925 04:42:50.088722 17442 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52505
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 04:42:50.106209 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37141
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:42:50.218909 17638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:42:50.219188 17638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:42:50.219231 17638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:42:50.223320 17638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:42:50.223390 17638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:42:50.223455 17638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 04:42:50.229166 17638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:42:50.230291 17638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:42:50.231338 17638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:42:50.241063 17643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:42:50.241040 17644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:42:50.241040 17646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:42:50.241420 17638 server_base.cc:1034] running on GCE node
I20260925 04:42:50.241923 17638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:42:50.242703 17638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:42:50.243896 17638 hybrid_clock.cc:648] HybridClock initialized: now 1790311370243870 us; error 43 us; skew 500 ppm
Sep 25 04:42:50 dist-test-slave-dc95 krb5kdc[17397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311370, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:42:50.246783 17638 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 04:42:50.247802 17638 webserver.cc:460] Webserver started at http://127.0.98.2:39239/ using document root <none> and password file <none>
I20260925 04:42:50.248364 17638 fs_manager.cc:359] Metadata directory not provided
I20260925 04:42:50.248431 17638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:42:50.248636 17638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:42:50.250741 17638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/minicluster-data/ts-1/data/instance:
uuid: "38a603009b0f472bacee9c3c6f33c2b3"
format_stamp: "Formatted at 2026-09-25 04:42:50 on dist-test-slave-dc95"
server_key: "c97b80ff0fe1e897354db77e93b2e8ef"
server_key_iv: "59c56dc4d2faaba2238c0b278f4ba40b"
server_key_version: "encryptionkey@0"
I20260925 04:42:50.251312 17638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/minicluster-data/ts-1/wal/instance:
uuid: "38a603009b0f472bacee9c3c6f33c2b3"
format_stamp: "Formatted at 2026-09-25 04:42:50 on dist-test-slave-dc95"
server_key: "c97b80ff0fe1e897354db77e93b2e8ef"
server_key_iv: "59c56dc4d2faaba2238c0b278f4ba40b"
server_key_version: "encryptionkey@0"
I20260925 04:42:50.255227 17638 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 04:42:50.258013 17653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:50.259474 17638 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:42:50.259649 17638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/minicluster-data/ts-1/wal
uuid: "38a603009b0f472bacee9c3c6f33c2b3"
format_stamp: "Formatted at 2026-09-25 04:42:50 on dist-test-slave-dc95"
server_key: "c97b80ff0fe1e897354db77e93b2e8ef"
server_key_iv: "59c56dc4d2faaba2238c0b278f4ba40b"
server_key_version: "encryptionkey@0"
I20260925 04:42:50.259770 17638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:42:50.277717 17638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:42:50.281472 17638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:42:50.281687 17638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:42:50.282321 17638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:42:50.283223 17638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:42:50.283272 17638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:50.283326 17638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:42:50.283354 17638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:50.292073 17638 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32997
I20260925 04:42:50.292090 17766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32997 every 8 connection(s)
I20260925 04:42:50.293041 17638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 04:42:50 dist-test-slave-dc95 krb5kdc[17397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311370, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:42:50.303025 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 17638
I20260925 04:42:50.303139 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/minicluster-data/ts-1/wal/instance
I20260925 04:42:50.303396 392 external_mini_cluster.cc:1428] Setting key e351aad525cbc2bd1f679d54b998c2c5
I20260925 04:42:50.307361 17481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:50.295248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33297 (local address 127.0.98.62:37141)
0925 04:42:50.295427 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:50.295431 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:50.296364 (+ 933us) server_negotiation.cc:407] Connection header received
0925 04:42:50.297450 (+ 1086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:50.297454 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:50.297512 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:50.297583 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:42:50.298909 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:50.299439 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:50.300656 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:50.300881 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:50.300973 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:50.303636 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:42:50.303641 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:42:50.303644 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:42:50.303674 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:42:50.305175 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:42:50.305781 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:42:50.305784 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:42:50.305785 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:42:50.305857 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:42:50.306191 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:42:50.306193 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:42:50.306194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:42:50.306364 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:42:50.306438 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:50.307046 (+ 608us) server_negotiation.cc:299] Negotiation successful
0925 04:42:50.307194 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260925 04:42:50.308228 17769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:50.295695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37141 (local address 127.0.98.2:33297)
0925 04:42:50.296183 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:50.296237 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 04:42:50.297203 (+ 966us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:42:50.297805 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:42:50.297813 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:42:50.298225 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:42:50.298754 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:50.298770 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:50.299564 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:50.299567 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:50.300517 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:50.300529 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:50.301037 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:50.301041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:50.301205 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:50.301810 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:42:50.301827 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:42:50.303404 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:42:50.305312 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:42:50.305320 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:42:50.305339 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:42:50.305690 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:42:50.305931 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:42:50.305933 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:42:50.305935 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:42:50.306113 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:42:50.306465 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:42:50.306473 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:42:50.306802 (+ 329us) client_negotiation.cc:769] Sending connection context
0925 04:42:50.306976 (+ 174us) client_negotiation.cc:240] Negotiation successful
0925 04:42:50.307275 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":121,"threads_started":1}
I20260925 04:42:50.309490 17767 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37141
I20260925 04:42:50.309731 17767 heartbeater.cc:464] Registering TS with master...
I20260925 04:42:50.310231 17767 heartbeater.cc:511] Master 127.0.98.62:37141 requested a full tablet report, sending...
I20260925 04:42:50.311329 17442 ts_manager.cc:194] Registered new tserver with Master: 38a603009b0f472bacee9c3c6f33c2b3 (127.0.98.2:32997)
I20260925 04:42:50.311959 17442 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33297
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 04:42:50.331068 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37141
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:42:50.443181 17772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:42:50.443481 17772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:42:50.443549 17772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:42:50.447738 17772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:42:50.447810 17772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:42:50.447894 17772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 04:42:50.453398 17772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17772
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:42:50.454540 17772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:42:50.455592 17772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:42:50.465615 17777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:42:50.465615 17780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:42:50.465677 17778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:42:50.465966 17772 server_base.cc:1034] running on GCE node
I20260925 04:42:50.466372 17772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:42:50.467012 17772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:42:50.468216 17772 hybrid_clock.cc:648] HybridClock initialized: now 1790311370468196 us; error 44 us; skew 500 ppm
Sep 25 04:42:50 dist-test-slave-dc95 krb5kdc[17397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311370, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:42:50.471024 17772 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 04:42:50.472023 17772 webserver.cc:460] Webserver started at http://127.0.98.3:36955/ using document root <none> and password file <none>
I20260925 04:42:50.472570 17772 fs_manager.cc:359] Metadata directory not provided
I20260925 04:42:50.472635 17772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:42:50.472824 17772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:42:50.474936 17772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/minicluster-data/ts-2/data/instance:
uuid: "9ea902dacd034a12a97f42310b1b2ed7"
format_stamp: "Formatted at 2026-09-25 04:42:50 on dist-test-slave-dc95"
server_key: "6c97683a5fb8175374c1329b752f2cb0"
server_key_iv: "4930ffcb370b5541392cb6867a0e0821"
server_key_version: "encryptionkey@0"
I20260925 04:42:50.475502 17772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9ea902dacd034a12a97f42310b1b2ed7"
format_stamp: "Formatted at 2026-09-25 04:42:50 on dist-test-slave-dc95"
server_key: "6c97683a5fb8175374c1329b752f2cb0"
server_key_iv: "4930ffcb370b5541392cb6867a0e0821"
server_key_version: "encryptionkey@0"
I20260925 04:42:50.479391 17772 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260925 04:42:50.482335 17787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:50.483770 17772 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:42:50.483897 17772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/minicluster-data/ts-2/wal
uuid: "9ea902dacd034a12a97f42310b1b2ed7"
format_stamp: "Formatted at 2026-09-25 04:42:50 on dist-test-slave-dc95"
server_key: "6c97683a5fb8175374c1329b752f2cb0"
server_key_iv: "4930ffcb370b5541392cb6867a0e0821"
server_key_version: "encryptionkey@0"
I20260925 04:42:50.484011 17772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:42:50.494066 17772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:42:50.497861 17772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:42:50.498071 17772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:42:50.498930 17772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:42:50.499903 17772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:42:50.499957 17772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:50.500015 17772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:42:50.500043 17772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:50.509088 17772 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43923
I20260925 04:42:50.509123 17900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43923 every 8 connection(s)
I20260925 04:42:50.510015 17772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
I20260925 04:42:50.517695 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 17772
I20260925 04:42:50.517814 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/minicluster-data/ts-2/wal/instance
I20260925 04:42:50.518095 392 external_mini_cluster.cc:1428] Setting key 46bd421075923d795eeb18b15f05069a
Sep 25 04:42:50 dist-test-slave-dc95 krb5kdc[17397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311370, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:42:50.523912 17481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:50.511951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37627 (local address 127.0.98.62:37141)
0925 04:42:50.512113 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:50.512117 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:50.513017 (+ 900us) server_negotiation.cc:407] Connection header received
0925 04:42:50.514025 (+ 1008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:50.514029 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:50.514086 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:50.514162 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:42:50.515423 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:50.516014 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:50.517192 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:50.517450 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:50.517559 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:50.520262 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:42:50.520266 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:42:50.520268 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:42:50.520299 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:42:50.521730 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:42:50.522283 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:42:50.522286 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:42:50.522288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:42:50.522373 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:42:50.522704 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:42:50.522707 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:42:50.522708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:42:50.522922 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:42:50.523015 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:50.523568 (+ 553us) server_negotiation.cc:299] Negotiation successful
0925 04:42:50.523745 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 04:42:50.524832 17903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:50.512318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37141 (local address 127.0.98.3:37627)
0925 04:42:50.512817 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:50.512864 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 04:42:50.513775 (+ 911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:42:50.514351 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:42:50.514359 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:42:50.514765 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:42:50.515260 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:50.515275 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:50.516140 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:50.516143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:50.517058 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:50.517069 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:50.517634 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:50.517638 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:50.517829 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:50.518506 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:42:50.518528 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:42:50.520082 (+ 1554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:42:50.521862 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:42:50.521868 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:42:50.521883 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:42:50.522179 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:42:50.522448 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:42:50.522450 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:42:50.522452 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:42:50.522599 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:42:50.523031 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:42:50.523037 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:42:50.523306 (+ 269us) client_negotiation.cc:769] Sending connection context
0925 04:42:50.523478 (+ 172us) client_negotiation.cc:240] Negotiation successful
0925 04:42:50.523960 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":120,"threads_started":1}
I20260925 04:42:50.526129 17901 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37141
I20260925 04:42:50.526381 17901 heartbeater.cc:464] Registering TS with master...
I20260925 04:42:50.526988 17901 heartbeater.cc:511] Master 127.0.98.62:37141 requested a full tablet report, sending...
I20260925 04:42:50.528200 17442 ts_manager.cc:194] Registered new tserver with Master: 9ea902dacd034a12a97f42310b1b2ed7 (127.0.98.3:43923)
I20260925 04:42:50.529003 17442 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37627
I20260925 04:42:50.532399 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:42:50.541777 17481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:50.534360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59744 (local address 127.0.98.62:37141)
0925 04:42:50.534536 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:50.534540 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:50.534578 (+ 38us) server_negotiation.cc:407] Connection header received
0925 04:42:50.534688 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:50.534690 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:50.534734 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:50.534806 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:42:50.535507 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:50.536127 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:50.537006 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:50.537238 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:50.537328 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:50.538739 (+ 1411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:42:50.538742 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:42:50.538744 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:42:50.538778 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:42:50.540069 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:42:50.540505 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:42:50.540508 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:42:50.540509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:42:50.540589 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:42:50.540833 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:42:50.540837 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:42:50.540838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:42:50.541113 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:42:50.541247 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:50.541492 (+ 245us) server_negotiation.cc:299] Negotiation successful
0925 04:42:50.541642 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260925 04:42:50.545063 17442 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59744:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:42:50.548194 17442 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:42:50.563278 17913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:50.557530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47582 (local address 127.0.98.3:43923)
0925 04:42:50.558526 (+ 996us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:50.558539 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:50.558564 (+ 25us) server_negotiation.cc:407] Connection header received
0925 04:42:50.558642 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:50.558648 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:50.558826 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:50.558970 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:42:50.559292 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:50.560338 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:50.561885 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:50.562495 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:50.562606 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:50.562785 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:50.562948 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 04:42:50.563111 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":877,"thread_start_us":157,"threads_started":1}
I20260925 04:42:50.563270 17912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:50.557383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43923 (local address 127.0.0.1:47582)
0925 04:42:50.557741 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:50.557798 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 04:42:50.557935 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:42:50.558970 (+ 1035us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:42:50.558976 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:42:50.558996 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:42:50.559165 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:50.559173 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:50.560506 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:50.560509 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:50.561762 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:50.561771 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:50.562616 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:50.562620 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:50.562743 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:50.562790 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 04:42:50.562909 (+ 119us) client_negotiation.cc:240] Negotiation successful
0925 04:42:50.563107 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":89,"threads_started":1}
I20260925 04:42:50.564009 17914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:50.557592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35915 (local address 127.0.0.1:37844)
0925 04:42:50.558192 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:50.558204 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:42:50.558278 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:42:50.559191 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:42:50.559194 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:42:50.559210 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:42:50.559293 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:50.559299 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:50.560922 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:50.560926 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:50.562328 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:50.562340 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:50.563579 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:50.563583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:50.563696 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:50.563721 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 04:42:50.563797 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 04:42:50.563869 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"mutex_wait_us":24,"thread_start_us":90,"threads_started":1}
I20260925 04:42:50.564200 17916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:50.558243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37844 (local address 127.0.98.1:35915)
0925 04:42:50.558759 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:50.558772 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:50.558792 (+ 20us) server_negotiation.cc:407] Connection header received
0925 04:42:50.558864 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:50.558869 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:50.559059 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:50.559219 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:42:50.559365 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:50.560762 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:50.562483 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:50.563265 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:50.563371 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:50.563512 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:50.563856 (+ 344us) server_negotiation.cc:299] Negotiation successful
0925 04:42:50.563966 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":102,"threads_started":1}
I20260925 04:42:50.565059 17915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:50.558150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32997 (local address 127.0.0.1:39660)
0925 04:42:50.558433 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:50.558445 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:42:50.558549 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:42:50.559547 (+ 998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:42:50.559550 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:42:50.559568 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:42:50.559697 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:50.559704 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:50.561098 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:50.561101 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:50.562387 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:50.562395 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:50.563972 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:50.563975 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:50.564744 (+ 769us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:50.564771 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 04:42:50.564844 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 04:42:50.564923 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":3,"thread_start_us":87,"threads_started":1}
I20260925 04:42:50.565867 17917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:50.558740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39660 (local address 127.0.98.2:32997)
0925 04:42:50.559149 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:50.559159 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:50.559179 (+ 20us) server_negotiation.cc:407] Connection header received
0925 04:42:50.559245 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:50.559250 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:50.559410 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:50.559553 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:42:50.559790 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:50.560960 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:50.562570 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:50.563838 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:50.565418 (+ 1580us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:50.565580 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:50.565631 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 04:42:50.565723 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":131,"threads_started":1}
I20260925 04:42:50.566591 17835 tablet_service.cc:1467] Processing CreateTablet for tablet 7519a860e3774810933e1303eaa65fa8 (DEFAULT_TABLE table=test-table [id=11538eeec8a646bc8a1cf005a495cd49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:42:50.566975 17567 tablet_service.cc:1467] Processing CreateTablet for tablet 7519a860e3774810933e1303eaa65fa8 (DEFAULT_TABLE table=test-table [id=11538eeec8a646bc8a1cf005a495cd49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:42:50.567840 17835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7519a860e3774810933e1303eaa65fa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:42:50.567945 17567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7519a860e3774810933e1303eaa65fa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:42:50.568792 17701 tablet_service.cc:1467] Processing CreateTablet for tablet 7519a860e3774810933e1303eaa65fa8 (DEFAULT_TABLE table=test-table [id=11538eeec8a646bc8a1cf005a495cd49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:42:50.569885 17701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7519a860e3774810933e1303eaa65fa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:42:50.574657 17918 tablet_bootstrap.cc:492] T 7519a860e3774810933e1303eaa65fa8 P 6d166e6940f844a1a6aadc712c6bb596: Bootstrap starting.
I20260925 04:42:50.575990 17919 tablet_bootstrap.cc:492] T 7519a860e3774810933e1303eaa65fa8 P 9ea902dacd034a12a97f42310b1b2ed7: Bootstrap starting.
I20260925 04:42:50.577023 17920 tablet_bootstrap.cc:492] T 7519a860e3774810933e1303eaa65fa8 P 38a603009b0f472bacee9c3c6f33c2b3: Bootstrap starting.
I20260925 04:42:50.577293 17918 tablet_bootstrap.cc:654] T 7519a860e3774810933e1303eaa65fa8 P 6d166e6940f844a1a6aadc712c6bb596: Neither blocks nor log segments found. Creating new log.
I20260925 04:42:50.578284 17919 tablet_bootstrap.cc:654] T 7519a860e3774810933e1303eaa65fa8 P 9ea902dacd034a12a97f42310b1b2ed7: Neither blocks nor log segments found. Creating new log.
I20260925 04:42:50.578601 17918 log.cc:824] T 7519a860e3774810933e1303eaa65fa8 P 6d166e6940f844a1a6aadc712c6bb596: Log is configured to *not* fsync() on all Append() calls
I20260925 04:42:50.579396 17920 tablet_bootstrap.cc:654] T 7519a860e3774810933e1303eaa65fa8 P 38a603009b0f472bacee9c3c6f33c2b3: Neither blocks nor log segments found. Creating new log.
I20260925 04:42:50.579465 17919 log.cc:824] T 7519a860e3774810933e1303eaa65fa8 P 9ea902dacd034a12a97f42310b1b2ed7: Log is configured to *not* fsync() on all Append() calls
I20260925 04:42:50.580230 17920 log.cc:824] T 7519a860e3774810933e1303eaa65fa8 P 38a603009b0f472bacee9c3c6f33c2b3: Log is configured to *not* fsync() on all Append() calls
I20260925 04:42:50.581022 17918 tablet_bootstrap.cc:492] T 7519a860e3774810933e1303eaa65fa8 P 6d166e6940f844a1a6aadc712c6bb596: No bootstrap required, opened a new log
I20260925 04:42:50.581254 17918 ts_tablet_manager.cc:1397] T 7519a860e3774810933e1303eaa65fa8 P 6d166e6940f844a1a6aadc712c6bb596: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260925 04:42:50.581424 17919 tablet_bootstrap.cc:492] T 7519a860e3774810933e1303eaa65fa8 P 9ea902dacd034a12a97f42310b1b2ed7: No bootstrap required, opened a new log
I20260925 04:42:50.581634 17919 ts_tablet_manager.cc:1397] T 7519a860e3774810933e1303eaa65fa8 P 9ea902dacd034a12a97f42310b1b2ed7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 04:42:50.582166 17920 tablet_bootstrap.cc:492] T 7519a860e3774810933e1303eaa65fa8 P 38a603009b0f472bacee9c3c6f33c2b3: No bootstrap required, opened a new log
I20260925 04:42:50.582340 17920 ts_tablet_manager.cc:1397] T 7519a860e3774810933e1303eaa65fa8 P 38a603009b0f472bacee9c3c6f33c2b3: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 04:42:50.584944 17919 raft_consensus.cc:348] T 7519a860e3774810933e1303eaa65fa8 P 9ea902dacd034a12a97f42310b1b2ed7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea902dacd034a12a97f42310b1b2ed7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43923 } } peers { permanent_uuid: "38a603009b0f472bacee9c3c6f33c2b3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32997 } } peers { permanent_uuid: "6d166e6940f844a1a6aadc712c6bb596" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35915 } }
I20260925 04:42:50.585235 17919 raft_consensus.cc:374] T 7519a860e3774810933e1303eaa65fa8 P 9ea902dacd034a12a97f42310b1b2ed7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:42:50.585289 17919 raft_consensus.cc:729] T 7519a860e3774810933e1303eaa65fa8 P 9ea902dacd034a12a97f42310b1b2ed7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ea902dacd034a12a97f42310b1b2ed7, State: Initialized, Role: FOLLOWER
I20260925 04:42:50.585294 17918 raft_consensus.cc:348] T 7519a860e3774810933e1303eaa65fa8 P 6d166e6940f844a1a6aadc712c6bb596 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea902dacd034a12a97f42310b1b2ed7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43923 } } peers { permanent_uuid: "38a603009b0f472bacee9c3c6f33c2b3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32997 } } peers { permanent_uuid: "6d166e6940f844a1a6aadc712c6bb596" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35915 } }
I20260925 04:42:50.585558 17918 raft_consensus.cc:374] T 7519a860e3774810933e1303eaa65fa8 P 6d166e6940f844a1a6aadc712c6bb596 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:42:50.585614 17918 raft_consensus.cc:729] T 7519a860e3774810933e1303eaa65fa8 P 6d166e6940f844a1a6aadc712c6bb596 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d166e6940f844a1a6aadc712c6bb596, State: Initialized, Role: FOLLOWER
I20260925 04:42:50.585785 17919 consensus_queue.cc:260] T 7519a860e3774810933e1303eaa65fa8 P 9ea902dacd034a12a97f42310b1b2ed7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea902dacd034a12a97f42310b1b2ed7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43923 } } peers { permanent_uuid: "38a603009b0f472bacee9c3c6f33c2b3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32997 } } peers { permanent_uuid: "6d166e6940f844a1a6aadc712c6bb596" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35915 } }
I20260925 04:42:50.586048 17918 consensus_queue.cc:260] T 7519a860e3774810933e1303eaa65fa8 P 6d166e6940f844a1a6aadc712c6bb596 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea902dacd034a12a97f42310b1b2ed7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43923 } } peers { permanent_uuid: "38a603009b0f472bacee9c3c6f33c2b3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32997 } } peers { permanent_uuid: "6d166e6940f844a1a6aadc712c6bb596" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35915 } }
I20260925 04:42:50.586162 17920 raft_consensus.cc:348] T 7519a860e3774810933e1303eaa65fa8 P 38a603009b0f472bacee9c3c6f33c2b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea902dacd034a12a97f42310b1b2ed7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43923 } } peers { permanent_uuid: "38a603009b0f472bacee9c3c6f33c2b3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32997 } } peers { permanent_uuid: "6d166e6940f844a1a6aadc712c6bb596" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35915 } }
I20260925 04:42:50.586428 17920 raft_consensus.cc:374] T 7519a860e3774810933e1303eaa65fa8 P 38a603009b0f472bacee9c3c6f33c2b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:42:50.586484 17920 raft_consensus.cc:729] T 7519a860e3774810933e1303eaa65fa8 P 38a603009b0f472bacee9c3c6f33c2b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38a603009b0f472bacee9c3c6f33c2b3, State: Initialized, Role: FOLLOWER
I20260925 04:42:50.586668 17901 heartbeater.cc:503] Master 127.0.98.62:37141 was elected leader, sending a full tablet report...
I20260925 04:42:50.586992 17920 consensus_queue.cc:260] T 7519a860e3774810933e1303eaa65fa8 P 38a603009b0f472bacee9c3c6f33c2b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea902dacd034a12a97f42310b1b2ed7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43923 } } peers { permanent_uuid: "38a603009b0f472bacee9c3c6f33c2b3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32997 } } peers { permanent_uuid: "6d166e6940f844a1a6aadc712c6bb596" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35915 } }
I20260925 04:42:50.587169 17918 ts_tablet_manager.cc:1428] T 7519a860e3774810933e1303eaa65fa8 P 6d166e6940f844a1a6aadc712c6bb596: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 04:42:50.587502 17633 heartbeater.cc:503] Master 127.0.98.62:37141 was elected leader, sending a full tablet report...
I20260925 04:42:50.588070 17919 ts_tablet_manager.cc:1428] T 7519a860e3774810933e1303eaa65fa8 P 9ea902dacd034a12a97f42310b1b2ed7: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 04:42:50.588140 17920 ts_tablet_manager.cc:1428] T 7519a860e3774810933e1303eaa65fa8 P 38a603009b0f472bacee9c3c6f33c2b3: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 04:42:50.588456 17767 heartbeater.cc:503] Master 127.0.98.62:37141 was elected leader, sending a full tablet report...
I20260925 04:42:50.751534 17926 raft_consensus.cc:482] T 7519a860e3774810933e1303eaa65fa8 P 38a603009b0f472bacee9c3c6f33c2b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:42:50.751775 17926 raft_consensus.cc:504] T 7519a860e3774810933e1303eaa65fa8 P 38a603009b0f472bacee9c3c6f33c2b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea902dacd034a12a97f42310b1b2ed7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43923 } } peers { permanent_uuid: "38a603009b0f472bacee9c3c6f33c2b3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32997 } } peers { permanent_uuid: "6d166e6940f844a1a6aadc712c6bb596" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35915 } }
I20260925 04:42:50.752866 17926 leader_election.cc:288] T 7519a860e3774810933e1303eaa65fa8 P 38a603009b0f472bacee9c3c6f33c2b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ea902dacd034a12a97f42310b1b2ed7 (127.0.98.3:43923), 6d166e6940f844a1a6aadc712c6bb596 (127.0.98.1:35915)
I20260925 04:42:50.757557 17769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:50.753159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43923 (local address 127.0.98.2:55855)
0925 04:42:50.753267 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:50.753282 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:42:50.753375 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:42:50.753714 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:42:50.753717 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:42:50.753742 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:42:50.753866 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:50.753904 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:50.754844 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:50.754847 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:50.756113 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:50.756125 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:50.757073 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:50.757076 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:50.757193 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:50.757221 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 04:42:50.757312 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 04:42:50.757413 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260925 04:42:50.757603 17913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:50.753260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55855 (local address 127.0.98.3:43923)
0925 04:42:50.753391 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:50.753396 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:50.753411 (+ 15us) server_negotiation.cc:407] Connection header received
0925 04:42:50.753552 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:50.753555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:50.753607 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:50.753680 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:42:50.754010 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:50.754716 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:50.756255 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:50.756919 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:50.757049 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:50.757122 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:50.757351 (+ 229us) server_negotiation.cc:299] Negotiation successful
0925 04:42:50.757489 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 04:42:50.757639 17916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:50.753328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58389 (local address 127.0.98.1:35915)
0925 04:42:50.753453 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:50.753458 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:50.753940 (+ 482us) server_negotiation.cc:407] Connection header received
0925 04:42:50.754047 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:50.754050 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:50.754097 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:50.754175 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:42:50.754532 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:50.755112 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:50.756495 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:50.757073 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:50.757170 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:50.757223 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:50.757440 (+ 217us) server_negotiation.cc:299] Negotiation successful
0925 04:42:50.757535 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 04:42:50.758033 17927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:50.753159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35915 (local address 127.0.98.2:58389)
0925 04:42:50.753839 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:50.753852 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:42:50.753964 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:42:50.754243 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:42:50.754246 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:42:50.754263 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:42:50.754421 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:50.754427 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:50.755227 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:50.755231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:50.756371 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:50.756379 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:50.757188 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:50.757192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:50.757279 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:50.757300 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 04:42:50.757814 (+ 514us) client_negotiation.cc:240] Negotiation successful
0925 04:42:50.757895 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"mutex_wait_us":362,"thread_start_us":129,"threads_started":1}
I20260925 04:42:50.757999 17924 raft_consensus.cc:482] T 7519a860e3774810933e1303eaa65fa8 P 9ea902dacd034a12a97f42310b1b2ed7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:42:50.758180 17924 raft_consensus.cc:504] T 7519a860e3774810933e1303eaa65fa8 P 9ea902dacd034a12a97f42310b1b2ed7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea902dacd034a12a97f42310b1b2ed7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43923 } } peers { permanent_uuid: "38a603009b0f472bacee9c3c6f33c2b3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32997 } } peers { permanent_uuid: "6d166e6940f844a1a6aadc712c6bb596" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35915 } }
I20260925 04:42:50.758584 17855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7519a860e3774810933e1303eaa65fa8" candidate_uuid: "38a603009b0f472bacee9c3c6f33c2b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ea902dacd034a12a97f42310b1b2ed7" is_pre_election: true
I20260925 04:42:50.758599 17587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7519a860e3774810933e1303eaa65fa8" candidate_uuid: "38a603009b0f472bacee9c3c6f33c2b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d166e6940f844a1a6aadc712c6bb596" is_pre_election: true
I20260925 04:42:50.758809 17855 raft_consensus.cc:2454] T 7519a860e3774810933e1303eaa65fa8 P 9ea902dacd034a12a97f42310b1b2ed7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38a603009b0f472bacee9c3c6f33c2b3 in term 0.
I20260925 04:42:50.758850 17587 raft_consensus.cc:2454] T 7519a860e3774810933e1303eaa65fa8 P 6d166e6940f844a1a6aadc712c6bb596 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38a603009b0f472bacee9c3c6f33c2b3 in term 0.
I20260925 04:42:50.759326 17656 leader_election.cc:302] T 7519a860e3774810933e1303eaa65fa8 P 38a603009b0f472bacee9c3c6f33c2b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38a603009b0f472bacee9c3c6f33c2b3, 9ea902dacd034a12a97f42310b1b2ed7; no voters:
I20260925 04:42:50.759389 17924 leader_election.cc:288] T 7519a860e3774810933e1303eaa65fa8 P 9ea902dacd034a12a97f42310b1b2ed7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 38a603009b0f472bacee9c3c6f33c2b3 (127.0.98.2:32997), 6d166e6940f844a1a6aadc712c6bb596 (127.0.98.1:35915)
I20260925 04:42:50.759862 17926 raft_consensus.cc:2781] T 7519a860e3774810933e1303eaa65fa8 P 38a603009b0f472bacee9c3c6f33c2b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:42:50.759955 17926 raft_consensus.cc:482] T 7519a860e3774810933e1303eaa65fa8 P 38a603009b0f472bacee9c3c6f33c2b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:42:50.760186 17926 raft_consensus.cc:3037] T 7519a860e3774810933e1303eaa65fa8 P 38a603009b0f472bacee9c3c6f33c2b3 [term 0 FOLLOWER]: Advancing to term 1
W20260925 04:42:50.760915 17902 tablet.cc:2369] T 7519a860e3774810933e1303eaa65fa8 P 9ea902dacd034a12a97f42310b1b2ed7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:42:50.761461 17926 raft_consensus.cc:504] T 7519a860e3774810933e1303eaa65fa8 P 38a603009b0f472bacee9c3c6f33c2b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea902dacd034a12a97f42310b1b2ed7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43923 } } peers { permanent_uuid: "38a603009b0f472bacee9c3c6f33c2b3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32997 } } peers { permanent_uuid: "6d166e6940f844a1a6aadc712c6bb596" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35915 } }
I20260925 04:42:50.761823 17926 leader_election.cc:288] T 7519a860e3774810933e1303eaa65fa8 P 38a603009b0f472bacee9c3c6f33c2b3 [CANDIDATE]: Term 1 election: Requested vote from peers 9ea902dacd034a12a97f42310b1b2ed7 (127.0.98.3:43923), 6d166e6940f844a1a6aadc712c6bb596 (127.0.98.1:35915)
I20260925 04:42:50.762159 17855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7519a860e3774810933e1303eaa65fa8" candidate_uuid: "38a603009b0f472bacee9c3c6f33c2b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ea902dacd034a12a97f42310b1b2ed7"
I20260925 04:42:50.762286 17855 raft_consensus.cc:3037] T 7519a860e3774810933e1303eaa65fa8 P 9ea902dacd034a12a97f42310b1b2ed7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:42:50.762393 17587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7519a860e3774810933e1303eaa65fa8" candidate_uuid: "38a603009b0f472bacee9c3c6f33c2b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d166e6940f844a1a6aadc712c6bb596"
I20260925 04:42:50.762511 17587 raft_consensus.cc:3037] T 7519a860e3774810933e1303eaa65fa8 P 6d166e6940f844a1a6aadc712c6bb596 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:42:50.763690 17855 raft_consensus.cc:2454] T 7519a860e3774810933e1303eaa65fa8 P 9ea902dacd034a12a97f42310b1b2ed7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38a603009b0f472bacee9c3c6f33c2b3 in term 1.
I20260925 04:42:50.763773 17587 raft_consensus.cc:2454] T 7519a860e3774810933e1303eaa65fa8 P 6d166e6940f844a1a6aadc712c6bb596 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38a603009b0f472bacee9c3c6f33c2b3 in term 1.
I20260925 04:42:50.764497 17916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:50.759941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49669 (local address 127.0.98.1:35915)
0925 04:42:50.760050 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:50.760054 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:50.760284 (+ 230us) server_negotiation.cc:407] Connection header received
0925 04:42:50.760444 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:50.760447 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:50.760488 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:50.760547 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:42:50.760773 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:50.761440 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:50.763114 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:50.763734 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:50.763816 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:50.763894 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:50.764307 (+ 413us) server_negotiation.cc:299] Negotiation successful
0925 04:42:50.764384 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 04:42:50.764520 17928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:50.759598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35915 (local address 127.0.98.3:49669)
0925 04:42:50.760210 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:50.760222 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:42:50.760335 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:42:50.760563 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:42:50.760565 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:42:50.760580 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:42:50.760680 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:50.760685 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:50.761541 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:50.761544 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:50.762971 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:50.762984 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:50.763986 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:50.763990 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:50.764161 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:50.764205 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 04:42:50.764270 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 04:42:50.764350 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"mutex_wait_us":151,"thread_start_us":98,"threads_started":1}
I20260925 04:42:50.764560 17903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:50.759557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32997 (local address 127.0.98.3:51809)
0925 04:42:50.759771 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:50.759786 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:42:50.759866 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:42:50.760158 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:42:50.760161 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:42:50.760180 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:42:50.760282 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:50.760289 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:50.761564 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:50.761567 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:50.762980 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:50.762986 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:50.763996 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:50.763999 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:50.764199 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:50.764226 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 04:42:50.764302 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 04:42:50.764386 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":113}
I20260925 04:42:50.764555 17654 leader_election.cc:302] T 7519a860e3774810933e1303eaa65fa8 P 38a603009b0f472bacee9c3c6f33c2b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38a603009b0f472bacee9c3c6f33c2b3, 6d166e6940f844a1a6aadc712c6bb596; no voters:
I20260925 04:42:50.764860 17587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7519a860e3774810933e1303eaa65fa8" candidate_uuid: "9ea902dacd034a12a97f42310b1b2ed7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d166e6940f844a1a6aadc712c6bb596" is_pre_election: true
I20260925 04:42:50.764889 17926 raft_consensus.cc:2781] T 7519a860e3774810933e1303eaa65fa8 P 38a603009b0f472bacee9c3c6f33c2b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:42:50.765054 17587 raft_consensus.cc:2379] T 7519a860e3774810933e1303eaa65fa8 P 6d166e6940f844a1a6aadc712c6bb596 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9ea902dacd034a12a97f42310b1b2ed7 in current term 1: Already voted for candidate 38a603009b0f472bacee9c3c6f33c2b3 in this term.
I20260925 04:42:50.765199 17926 raft_consensus.cc:686] T 7519a860e3774810933e1303eaa65fa8 P 38a603009b0f472bacee9c3c6f33c2b3 [term 1 LEADER]: Becoming Leader. State: Replica: 38a603009b0f472bacee9c3c6f33c2b3, State: Running, Role: LEADER
I20260925 04:42:50.765589 17926 consensus_queue.cc:237] T 7519a860e3774810933e1303eaa65fa8 P 38a603009b0f472bacee9c3c6f33c2b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea902dacd034a12a97f42310b1b2ed7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43923 } } peers { permanent_uuid: "38a603009b0f472bacee9c3c6f33c2b3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32997 } } peers { permanent_uuid: "6d166e6940f844a1a6aadc712c6bb596" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35915 } }
I20260925 04:42:50.765703 17917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:50.759743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51809 (local address 127.0.98.2:32997)
0925 04:42:50.759840 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:50.759844 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:50.759856 (+ 12us) server_negotiation.cc:407] Connection header received
0925 04:42:50.759954 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:50.759957 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:50.759999 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:50.760062 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:42:50.760391 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:50.761091 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:50.763116 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:50.763771 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:50.763853 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:50.763906 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:50.765450 (+ 1544us) server_negotiation.cc:299] Negotiation successful
0925 04:42:50.765543 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 04:42:50.766301 17721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7519a860e3774810933e1303eaa65fa8" candidate_uuid: "9ea902dacd034a12a97f42310b1b2ed7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38a603009b0f472bacee9c3c6f33c2b3" is_pre_election: true
I20260925 04:42:50.766909 17790 leader_election.cc:302] T 7519a860e3774810933e1303eaa65fa8 P 9ea902dacd034a12a97f42310b1b2ed7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ea902dacd034a12a97f42310b1b2ed7; no voters: 38a603009b0f472bacee9c3c6f33c2b3, 6d166e6940f844a1a6aadc712c6bb596
I20260925 04:42:50.767160 17924 raft_consensus.cc:2726] T 7519a860e3774810933e1303eaa65fa8 P 9ea902dacd034a12a97f42310b1b2ed7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 04:42:50.768975 17442 catalog_manager.cc:5495] T 7519a860e3774810933e1303eaa65fa8 P 38a603009b0f472bacee9c3c6f33c2b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 38a603009b0f472bacee9c3c6f33c2b3 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "38a603009b0f472bacee9c3c6f33c2b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea902dacd034a12a97f42310b1b2ed7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38a603009b0f472bacee9c3c6f33c2b3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d166e6940f844a1a6aadc712c6bb596" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35915 } health_report { overall_health: UNKNOWN } } }
W20260925 04:42:50.794553 17768 tablet.cc:2369] T 7519a860e3774810933e1303eaa65fa8 P 38a603009b0f472bacee9c3c6f33c2b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:42:50.802278 17917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:50.798550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39676 (local address 127.0.98.2:32997)
0925 04:42:50.798711 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:50.798715 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:50.798729 (+ 14us) server_negotiation.cc:407] Connection header received
0925 04:42:50.798809 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:50.798811 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:50.798883 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:50.798981 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:42:50.799296 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:50.799927 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:50.800811 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:50.801038 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:50.801112 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:50.801420 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:42:50.801858 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:42:50.801949 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:50.802081 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 04:42:50.802138 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 04:42:50.808547 17587 raft_consensus.cc:1261] T 7519a860e3774810933e1303eaa65fa8 P 6d166e6940f844a1a6aadc712c6bb596 [term 1 FOLLOWER]: Refusing update from remote peer 38a603009b0f472bacee9c3c6f33c2b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:42:50.808560 17855 raft_consensus.cc:1261] T 7519a860e3774810933e1303eaa65fa8 P 9ea902dacd034a12a97f42310b1b2ed7 [term 1 FOLLOWER]: Refusing update from remote peer 38a603009b0f472bacee9c3c6f33c2b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:42:50.809276 17929 consensus_queue.cc:1035] T 7519a860e3774810933e1303eaa65fa8 P 38a603009b0f472bacee9c3c6f33c2b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ea902dacd034a12a97f42310b1b2ed7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:42:50.809487 17926 consensus_queue.cc:1035] T 7519a860e3774810933e1303eaa65fa8 P 38a603009b0f472bacee9c3c6f33c2b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d166e6940f844a1a6aadc712c6bb596" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:42:50.816766 17932 mvcc.cc:205] Tried to move back new op lower bound from 7333115374823559168 to 7333115374659158016. Current Snapshot: MvccSnapshot[applied={T|T < 7333115374823559168}]
I20260925 04:42:50.817571 17933 mvcc.cc:205] Tried to move back new op lower bound from 7333115374823559168 to 7333115374659158016. Current Snapshot: MvccSnapshot[applied={T|T < 7333115374823559168}]
I20260925 04:42:50.827414 17441 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:42:50.827651 17441 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:42:50.830207 17441 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eab965905936434a915e5d3fc010dff7: Sending DeleteTablet for 3 replicas of tablet 7519a860e3774810933e1303eaa65fa8
I20260925 04:42:50.831048 17835 tablet_service.cc:1514] Processing DeleteTablet for tablet 7519a860e3774810933e1303eaa65fa8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:42:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47582
I20260925 04:42:50.831077 17567 tablet_service.cc:1514] Processing DeleteTablet for tablet 7519a860e3774810933e1303eaa65fa8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:42:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37844
I20260925 04:42:50.831573 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 17504
I20260925 04:42:50.831739 17701 tablet_service.cc:1514] Processing DeleteTablet for tablet 7519a860e3774810933e1303eaa65fa8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:42:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39660
I20260925 04:42:50.832722 17942 tablet_replica.cc:330] T 7519a860e3774810933e1303eaa65fa8 P 9ea902dacd034a12a97f42310b1b2ed7: stopping tablet replica
I20260925 04:42:50.833037 17942 raft_consensus.cc:2229] T 7519a860e3774810933e1303eaa65fa8 P 9ea902dacd034a12a97f42310b1b2ed7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:42:50.833374 17942 raft_consensus.cc:2258] T 7519a860e3774810933e1303eaa65fa8 P 9ea902dacd034a12a97f42310b1b2ed7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:42:50.833739 17943 tablet_replica.cc:330] T 7519a860e3774810933e1303eaa65fa8 P 38a603009b0f472bacee9c3c6f33c2b3: stopping tablet replica
I20260925 04:42:50.834000 17943 raft_consensus.cc:2229] T 7519a860e3774810933e1303eaa65fa8 P 38a603009b0f472bacee9c3c6f33c2b3 [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:42:50.834352 17943 raft_consensus.cc:2258] T 7519a860e3774810933e1303eaa65fa8 P 38a603009b0f472bacee9c3c6f33c2b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:42:50.835004 17942 ts_tablet_manager.cc:1905] T 7519a860e3774810933e1303eaa65fa8 P 9ea902dacd034a12a97f42310b1b2ed7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:42:50.835807 17943 ts_tablet_manager.cc:1905] T 7519a860e3774810933e1303eaa65fa8 P 38a603009b0f472bacee9c3c6f33c2b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:42:50.837797 17942 ts_tablet_manager.cc:1918] T 7519a860e3774810933e1303eaa65fa8 P 9ea902dacd034a12a97f42310b1b2ed7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:42:50.837896 17942 log.cc:1196] T 7519a860e3774810933e1303eaa65fa8 P 9ea902dacd034a12a97f42310b1b2ed7: Deleting WAL directory at /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/minicluster-data/ts-2/wal/wals/7519a860e3774810933e1303eaa65fa8
W20260925 04:42:50.838290 17426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35915 (error 108)
I20260925 04:42:50.838459 17942 ts_tablet_manager.cc:1939] T 7519a860e3774810933e1303eaa65fa8 P 9ea902dacd034a12a97f42310b1b2ed7: Deleting consensus metadata
I20260925 04:42:50.838905 17943 ts_tablet_manager.cc:1918] T 7519a860e3774810933e1303eaa65fa8 P 38a603009b0f472bacee9c3c6f33c2b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:42:50.839072 17943 log.cc:1196] T 7519a860e3774810933e1303eaa65fa8 P 38a603009b0f472bacee9c3c6f33c2b3: Deleting WAL directory at /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311123478858-392-0/minicluster-data/ts-1/wal/wals/7519a860e3774810933e1303eaa65fa8
I20260925 04:42:50.839128 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 17638
I20260925 04:42:50.839457 17943 ts_tablet_manager.cc:1939] T 7519a860e3774810933e1303eaa65fa8 P 38a603009b0f472bacee9c3c6f33c2b3: Deleting consensus metadata
I20260925 04:42:50.840444 17915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:50.840142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35915 (local address 127.0.0.1:37860)
0925 04:42:50.840276 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:50.840356 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 04:42:50.840677 17426 catalog_manager.cc:4584] TS 6d166e6940f844a1a6aadc712c6bb596 (127.0.98.1:35915): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7519a860e3774810933e1303eaa65fa8: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35915: connect: Connection refused (error 111)
I20260925 04:42:50.841238 17428 catalog_manager.cc:4841] TS 9ea902dacd034a12a97f42310b1b2ed7 (127.0.98.3:43923): tablet 7519a860e3774810933e1303eaa65fa8 (table test-table [id=11538eeec8a646bc8a1cf005a495cd49]) successfully deleted
I20260925 04:42:50.844619 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 17772
I20260925 04:42:50.845703 17915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:50.845426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32997 (local address 127.0.0.1:39690)
0925 04:42:50.845562 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:50.845619 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 04:42:50.849797 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 17410
2026-09-25T04:42:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2362 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-25T04:42:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:42:50Z Disabled control of system clock
I20260925 04:42:50.877561 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311123478858-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311123478858-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311123478858-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37235
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37235
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 04:42:50.986625 17948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:42:50.986914 17948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:42:50.986953 17948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:42:50.991077 17948 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:42:50.991155 17948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:42:50.991169 17948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:42:50.991186 17948 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:42:50.991200 17948 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:42:50.996263 17948 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311123478858-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311123478858-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37235
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311123478858-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:42:50.997376 17948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:42:50.998814 17948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 04:42:51.007522 17948 server_base.cc:1034] running on GCE node
W20260925 04:42:51.007471 17953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:42:51.007471 17954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:42:51.007472 17956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:42:51.008384 17948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:42:51.009497 17948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:42:51.010666 17948 hybrid_clock.cc:648] HybridClock initialized: now 1790311371010647 us; error 32 us; skew 500 ppm
I20260925 04:42:51.012735 17948 webserver.cc:460] Webserver started at http://127.0.98.62:34147/ using document root <none> and password file <none>
I20260925 04:42:51.013331 17948 fs_manager.cc:359] Metadata directory not provided
I20260925 04:42:51.013396 17948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:42:51.013588 17948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:42:51.015805 17948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311123478858-392-0/minicluster-data/master-0/data/instance:
uuid: "5e745bfc95c94f0694b43e01d761e52a"
format_stamp: "Formatted at 2026-09-25 04:42:51 on dist-test-slave-dc95"
server_key: "ee22b7836f6833520020105267d98f84"
server_key_iv: "1988b8fc159df556d843a97dca11b5fb"
server_key_version: "encryptionkey@0"
I20260925 04:42:51.016410 17948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311123478858-392-0/minicluster-data/master-0/wal/instance:
uuid: "5e745bfc95c94f0694b43e01d761e52a"
format_stamp: "Formatted at 2026-09-25 04:42:51 on dist-test-slave-dc95"
server_key: "ee22b7836f6833520020105267d98f84"
server_key_iv: "1988b8fc159df556d843a97dca11b5fb"
server_key_version: "encryptionkey@0"
I20260925 04:42:51.020776 17948 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:42:51.023499 17962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:51.024950 17948 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:42:51.025072 17948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311123478858-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311123478858-392-0/minicluster-data/master-0/wal
uuid: "5e745bfc95c94f0694b43e01d761e52a"
format_stamp: "Formatted at 2026-09-25 04:42:51 on dist-test-slave-dc95"
server_key: "ee22b7836f6833520020105267d98f84"
server_key_iv: "1988b8fc159df556d843a97dca11b5fb"
server_key_version: "encryptionkey@0"
I20260925 04:42:51.025216 17948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311123478858-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311123478858-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311123478858-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:42:51.044137 17948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:42:51.044881 17948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:42:51.045065 17948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:42:51.052318 17948 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37235
I20260925 04:42:51.052330 18014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37235 every 8 connection(s)
I20260925 04:42:51.053349 17948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
I20260925 04:42:51.056799 18015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:42:51.063220 18015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e745bfc95c94f0694b43e01d761e52a: Bootstrap starting.
I20260925 04:42:51.063457 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 17948
I20260925 04:42:51.063578 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311123478858-392-0/minicluster-data/master-0/wal/instance
I20260925 04:42:51.063911 392 external_mini_cluster.cc:1428] Setting key c4089da9454219782a0a3a784df3a5ae
I20260925 04:42:51.066366 18015 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e745bfc95c94f0694b43e01d761e52a: Neither blocks nor log segments found. Creating new log.
I20260925 04:42:51.067159 18015 log.cc:824] T 00000000000000000000000000000000 P 5e745bfc95c94f0694b43e01d761e52a: Log is configured to *not* fsync() on all Append() calls
I20260925 04:42:51.069378 18015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e745bfc95c94f0694b43e01d761e52a: No bootstrap required, opened a new log
I20260925 04:42:51.073086 18015 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e745bfc95c94f0694b43e01d761e52a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e745bfc95c94f0694b43e01d761e52a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37235 } }
I20260925 04:42:51.073351 18015 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e745bfc95c94f0694b43e01d761e52a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:42:51.073403 18015 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e745bfc95c94f0694b43e01d761e52a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e745bfc95c94f0694b43e01d761e52a, State: Initialized, Role: FOLLOWER
I20260925 04:42:51.073832 18018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:51.065420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49682 (local address 127.0.98.62:37235)
0925 04:42:51.066173 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:51.066190 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:51.066229 (+ 39us) server_negotiation.cc:407] Connection header received
0925 04:42:51.066989 (+ 760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:51.067009 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:51.067464 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:51.067890 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:42:51.068280 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:51.069376 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:51.070380 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:51.070884 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:51.071039 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:51.071909 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:42:51.071914 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:42:51.071930 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:42:51.072358 (+ 428us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:42:51.072419 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:42:51.072441 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:42:51.072543 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:51.072697 (+ 154us) server_negotiation.cc:299] Negotiation successful
0925 04:42:51.072986 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":138,"threads_started":1}
I20260925 04:42:51.073858 18015 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e745bfc95c94f0694b43e01d761e52a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e745bfc95c94f0694b43e01d761e52a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37235 } }
I20260925 04:42:51.074020 18015 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e745bfc95c94f0694b43e01d761e52a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:42:51.074079 18015 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e745bfc95c94f0694b43e01d761e52a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:42:51.074199 18015 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e745bfc95c94f0694b43e01d761e52a [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:42:51.075150 18015 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e745bfc95c94f0694b43e01d761e52a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e745bfc95c94f0694b43e01d761e52a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37235 } }
I20260925 04:42:51.075493 18015 leader_election.cc:302] T 00000000000000000000000000000000 P 5e745bfc95c94f0694b43e01d761e52a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e745bfc95c94f0694b43e01d761e52a; no voters:
I20260925 04:42:51.075851 18015 leader_election.cc:288] T 00000000000000000000000000000000 P 5e745bfc95c94f0694b43e01d761e52a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:42:51.076002 18020 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e745bfc95c94f0694b43e01d761e52a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:42:51.076242 18020 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e745bfc95c94f0694b43e01d761e52a [term 1 LEADER]: Becoming Leader. State: Replica: 5e745bfc95c94f0694b43e01d761e52a, State: Running, Role: LEADER
I20260925 04:42:51.076586 18020 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e745bfc95c94f0694b43e01d761e52a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e745bfc95c94f0694b43e01d761e52a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37235 } }
I20260925 04:42:51.077183 18015 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e745bfc95c94f0694b43e01d761e52a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:42:51.077826 18020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e745bfc95c94f0694b43e01d761e52a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e745bfc95c94f0694b43e01d761e52a. Latest consensus state: current_term: 1 leader_uuid: "5e745bfc95c94f0694b43e01d761e52a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e745bfc95c94f0694b43e01d761e52a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37235 } } }
I20260925 04:42:51.077952 18020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e745bfc95c94f0694b43e01d761e52a [sys.catalog]: This master's current role is: LEADER
I20260925 04:42:51.078342 18026 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:42:51.081502 18026 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:42:51.078341 18022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e745bfc95c94f0694b43e01d761e52a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e745bfc95c94f0694b43e01d761e52a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e745bfc95c94f0694b43e01d761e52a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37235 } } }
I20260925 04:42:51.083360 18022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e745bfc95c94f0694b43e01d761e52a [sys.catalog]: This master's current role is: LEADER
I20260925 04:42:51.087285 18026 catalog_manager.cc:1347] Generated new cluster ID: 8d499955773b4702afed9f61018d39fc
I20260925 04:42:51.087368 18026 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:42:51.110532 18026 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:42:51.111948 18026 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:42:51.117715 18026 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e745bfc95c94f0694b43e01d761e52a: Generated new TSK 0
I20260925 04:42:51.118630 18026 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:42:51.128965 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311123478858-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311123478858-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311123478858-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37235
--builtin_ntp_servers=127.0.98.20:37451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 04:42:51.244311 18039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:42:51.244609 18039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:42:51.244650 18039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:42:51.248874 18039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:42:51.248952 18039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:42:51.249010 18039 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 04:42:51.254594 18039 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311123478858-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311123478858-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311123478858-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:42:51.255829 18039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:42:51.257309 18039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:42:51.267892 18044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:42:51.268051 18039 server_base.cc:1034] running on GCE node
W20260925 04:42:51.267894 18047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:42:51.268293 18045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:42:51.268728 18039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:42:51.269418 18039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:42:51.270608 18039 hybrid_clock.cc:648] HybridClock initialized: now 1790311371270571 us; error 49 us; skew 500 ppm
I20260925 04:42:51.272842 18039 webserver.cc:460] Webserver started at http://127.0.98.1:38021/ using document root <none> and password file <none>
I20260925 04:42:51.273442 18039 fs_manager.cc:359] Metadata directory not provided
I20260925 04:42:51.273515 18039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:42:51.273710 18039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:42:51.275974 18039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311123478858-392-0/minicluster-data/ts-0/data/instance:
uuid: "1e580a9e4e5f4d74878558b8c31843b0"
format_stamp: "Formatted at 2026-09-25 04:42:51 on dist-test-slave-dc95"
server_key: "991a86d6a17e670d0b1e8b8f1901ff7f"
server_key_iv: "99f74fb31721331067de1240bb9c23cc"
server_key_version: "encryptionkey@0"
I20260925 04:42:51.276551 18039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311123478858-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1e580a9e4e5f4d74878558b8c31843b0"
format_stamp: "Formatted at 2026-09-25 04:42:51 on dist-test-slave-dc95"
server_key: "991a86d6a17e670d0b1e8b8f1901ff7f"
server_key_iv: "99f74fb31721331067de1240bb9c23cc"
server_key_version: "encryptionkey@0"
I20260925 04:42:51.280572 18039 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 04:42:51.283313 18053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:51.284704 18039 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:42:51.284849 18039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311123478858-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311123478858-392-0/minicluster-data/ts-0/wal
uuid: "1e580a9e4e5f4d74878558b8c31843b0"
format_stamp: "Formatted at 2026-09-25 04:42:51 on dist-test-slave-dc95"
server_key: "991a86d6a17e670d0b1e8b8f1901ff7f"
server_key_iv: "99f74fb31721331067de1240bb9c23cc"
server_key_version: "encryptionkey@0"
I20260925 04:42:51.284965 18039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311123478858-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311123478858-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311123478858-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:42:51.298861 18039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:42:51.299561 18039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:42:51.299769 18039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:42:51.300381 18039 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:42:51.301327 18039 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:42:51.301373 18039 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:51.301422 18039 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:42:51.301436 18039 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:51.309979 18039 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45405
I20260925 04:42:51.310048 18166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45405 every 8 connection(s)
I20260925 04:42:51.310904 18039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
I20260925 04:42:51.315834 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 18039
I20260925 04:42:51.315937 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311123478858-392-0/minicluster-data/ts-0/wal/instance
I20260925 04:42:51.316179 392 external_mini_cluster.cc:1428] Setting key b330acfc8b544d272134a1a5332bd555
I20260925 04:42:51.318142 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311123478858-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311123478858-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311123478858-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37235
--builtin_ntp_servers=127.0.98.20:37451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 04:42:51.320796 18018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:51.312769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36055 (local address 127.0.98.62:37235)
0925 04:42:51.312905 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:51.312908 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:51.313953 (+ 1045us) server_negotiation.cc:407] Connection header received
0925 04:42:51.315312 (+ 1359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:51.315314 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:51.315365 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:51.315441 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:42:51.316548 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:51.317179 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:51.318667 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:51.318894 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:51.318988 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:51.320254 (+ 1266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:42:51.320258 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:42:51.320261 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:42:51.320277 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:42:51.320301 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:42:51.320309 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:42:51.320374 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:51.320602 (+ 228us) server_negotiation.cc:299] Negotiation successful
0925 04:42:51.320668 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 04:42:51.321913 18169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:51.313129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37235 (local address 127.0.98.1:36055)
0925 04:42:51.313798 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:51.313840 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 04:42:51.315103 (+ 1263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:42:51.315637 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:42:51.315648 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:42:51.315748 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:42:51.316377 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:51.316395 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:51.317322 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:51.317326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:51.318536 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:51.318546 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:51.319055 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:51.319059 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:51.319227 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:51.319949 (+ 722us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:42:51.319951 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:42:51.319998 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:42:51.320075 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:42:51.320391 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:42:51.320403 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:42:51.320409 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 04:42:51.320544 (+ 135us) client_negotiation.cc:240] Negotiation successful
0925 04:42:51.320864 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":113,"threads_started":1}
I20260925 04:42:51.323421 18167 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37235
I20260925 04:42:51.323701 18167 heartbeater.cc:464] Registering TS with master...
I20260925 04:42:51.324301 18167 heartbeater.cc:511] Master 127.0.98.62:37235 requested a full tablet report, sending...
I20260925 04:42:51.326020 17979 ts_manager.cc:194] Registered new tserver with Master: 1e580a9e4e5f4d74878558b8c31843b0 (127.0.98.1:45405)
I20260925 04:42:51.327178 17979 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:36055
W20260925 04:42:51.436618 18170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:42:51.436914 18170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:42:51.436954 18170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:42:51.441216 18170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:42:51.441293 18170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:42:51.441354 18170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 04:42:51.446388 18170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311123478858-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311123478858-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311123478858-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:42:51.447561 18170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:42:51.449008 18170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:42:51.459064 18176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:42:51.459074 18178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:42:51.459074 18175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:42:51.459362 18170 server_base.cc:1034] running on GCE node
I20260925 04:42:51.459928 18170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:42:51.460625 18170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:42:51.461807 18170 hybrid_clock.cc:648] HybridClock initialized: now 1790311371461780 us; error 44 us; skew 500 ppm
I20260925 04:42:51.463944 18170 webserver.cc:460] Webserver started at http://127.0.98.2:41393/ using document root <none> and password file <none>
I20260925 04:42:51.464551 18170 fs_manager.cc:359] Metadata directory not provided
I20260925 04:42:51.464618 18170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:42:51.464818 18170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:42:51.466944 18170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311123478858-392-0/minicluster-data/ts-1/data/instance:
uuid: "de6310719e08466594f493248403cc38"
format_stamp: "Formatted at 2026-09-25 04:42:51 on dist-test-slave-dc95"
server_key: "c9e25bf0fe1ff09e92679b424f6c4074"
server_key_iv: "904909e1fe9b7f1e5beebb3e5a366622"
server_key_version: "encryptionkey@0"
I20260925 04:42:51.467499 18170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311123478858-392-0/minicluster-data/ts-1/wal/instance:
uuid: "de6310719e08466594f493248403cc38"
format_stamp: "Formatted at 2026-09-25 04:42:51 on dist-test-slave-dc95"
server_key: "c9e25bf0fe1ff09e92679b424f6c4074"
server_key_iv: "904909e1fe9b7f1e5beebb3e5a366622"
server_key_version: "encryptionkey@0"
I20260925 04:42:51.471778 18170 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260925 04:42:51.474645 18184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:51.476284 18170 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 04:42:51.476415 18170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311123478858-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311123478858-392-0/minicluster-data/ts-1/wal
uuid: "de6310719e08466594f493248403cc38"
format_stamp: "Formatted at 2026-09-25 04:42:51 on dist-test-slave-dc95"
server_key: "c9e25bf0fe1ff09e92679b424f6c4074"
server_key_iv: "904909e1fe9b7f1e5beebb3e5a366622"
server_key_version: "encryptionkey@0"
I20260925 04:42:51.476527 18170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311123478858-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311123478858-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311123478858-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:42:51.512086 18170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:42:51.512871 18170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:42:51.513041 18170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:42:51.513655 18170 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:42:51.514654 18170 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:42:51.514710 18170 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:51.514751 18170 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:42:51.514778 18170 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:51.523629 18170 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41741
I20260925 04:42:51.523667 18297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41741 every 8 connection(s)
I20260925 04:42:51.524595 18170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
I20260925 04:42:51.525733 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 18170
I20260925 04:42:51.525835 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311123478858-392-0/minicluster-data/ts-1/wal/instance
I20260925 04:42:51.526091 392 external_mini_cluster.cc:1428] Setting key e3c871dad435dab4b84db16865466a5e
I20260925 04:42:51.528153 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311123478858-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311123478858-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311123478858-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37235
--builtin_ntp_servers=127.0.98.20:37451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 04:42:51.535581 18018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:51.526526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35567 (local address 127.0.98.62:37235)
0925 04:42:51.526673 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:51.526677 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:51.527935 (+ 1258us) server_negotiation.cc:407] Connection header received
0925 04:42:51.529748 (+ 1813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:51.529752 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:51.529840 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:51.529938 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:42:51.530936 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:51.531553 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:51.533336 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:51.533564 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:51.533686 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:51.535082 (+ 1396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:42:51.535086 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:42:51.535088 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:42:51.535104 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:42:51.535127 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:42:51.535136 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:42:51.535198 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:51.535381 (+ 183us) server_negotiation.cc:299] Negotiation successful
0925 04:42:51.535452 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 04:42:51.536906 18300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:51.526989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37235 (local address 127.0.98.2:35567)
0925 04:42:51.527766 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:51.527824 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 04:42:51.529455 (+ 1631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:42:51.530092 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:42:51.530100 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:42:51.530196 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:42:51.530762 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:51.530782 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:51.531711 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:51.531715 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:51.533179 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:51.533195 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:51.533764 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:51.533769 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:51.534004 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:51.534714 (+ 710us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:42:51.534716 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:42:51.534784 (+ 68us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:42:51.534889 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:42:51.535221 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:42:51.535231 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:42:51.535238 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 04:42:51.535383 (+ 145us) client_negotiation.cc:240] Negotiation successful
0925 04:42:51.535776 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":110,"threads_started":1}
I20260925 04:42:51.538249 18298 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37235
I20260925 04:42:51.538502 18298 heartbeater.cc:464] Registering TS with master...
I20260925 04:42:51.539098 18298 heartbeater.cc:511] Master 127.0.98.62:37235 requested a full tablet report, sending...
I20260925 04:42:51.540172 17979 ts_manager.cc:194] Registered new tserver with Master: de6310719e08466594f493248403cc38 (127.0.98.2:41741)
I20260925 04:42:51.540755 17979 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:35567
W20260925 04:42:51.646698 18301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:42:51.647017 18301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:42:51.647076 18301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:42:51.651291 18301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:42:51.651378 18301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:42:51.651455 18301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 04:42:51.656610 18301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311123478858-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311123478858-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311123478858-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:42:51.657711 18301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:42:51.659130 18301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:42:51.669397 18307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:42:51.669494 18301 server_base.cc:1034] running on GCE node
W20260925 04:42:51.669399 18309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:42:51.669415 18306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:42:51.670105 18301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:42:51.670817 18301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:42:51.672039 18301 hybrid_clock.cc:648] HybridClock initialized: now 1790311371672022 us; error 33 us; skew 500 ppm
I20260925 04:42:51.674134 18301 webserver.cc:460] Webserver started at http://127.0.98.3:45465/ using document root <none> and password file <none>
I20260925 04:42:51.674731 18301 fs_manager.cc:359] Metadata directory not provided
I20260925 04:42:51.674799 18301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:42:51.675009 18301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:42:51.677266 18301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311123478858-392-0/minicluster-data/ts-2/data/instance:
uuid: "bcf0806bee8b44ecbf5528701baaa063"
format_stamp: "Formatted at 2026-09-25 04:42:51 on dist-test-slave-dc95"
server_key: "cfd6028363177388f12ff2fcb160537d"
server_key_iv: "027c9d441052dab55bb0f3b7c5682efe"
server_key_version: "encryptionkey@0"
I20260925 04:42:51.677815 18301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311123478858-392-0/minicluster-data/ts-2/wal/instance:
uuid: "bcf0806bee8b44ecbf5528701baaa063"
format_stamp: "Formatted at 2026-09-25 04:42:51 on dist-test-slave-dc95"
server_key: "cfd6028363177388f12ff2fcb160537d"
server_key_iv: "027c9d441052dab55bb0f3b7c5682efe"
server_key_version: "encryptionkey@0"
I20260925 04:42:51.681963 18301 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 04:42:51.684763 18315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:51.686183 18301 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:42:51.686326 18301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311123478858-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311123478858-392-0/minicluster-data/ts-2/wal
uuid: "bcf0806bee8b44ecbf5528701baaa063"
format_stamp: "Formatted at 2026-09-25 04:42:51 on dist-test-slave-dc95"
server_key: "cfd6028363177388f12ff2fcb160537d"
server_key_iv: "027c9d441052dab55bb0f3b7c5682efe"
server_key_version: "encryptionkey@0"
I20260925 04:42:51.686452 18301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311123478858-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311123478858-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311123478858-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:42:51.701092 18301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:42:51.701828 18301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:42:51.702008 18301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:42:51.702605 18301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:42:51.703563 18301 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:42:51.703634 18301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:51.703689 18301 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:42:51.703716 18301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:51.712220 18301 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33311
I20260925 04:42:51.712245 18428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33311 every 8 connection(s)
I20260925 04:42:51.713153 18301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
I20260925 04:42:51.715263 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 18301
I20260925 04:42:51.715361 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311123478858-392-0/minicluster-data/ts-2/wal/instance
I20260925 04:42:51.715641 392 external_mini_cluster.cc:1428] Setting key e5fc28a9493d59a2db05d8d69b4a7957
I20260925 04:42:51.722966 18018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:51.715064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48323 (local address 127.0.98.62:37235)
0925 04:42:51.715221 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:51.715227 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:51.716453 (+ 1226us) server_negotiation.cc:407] Connection header received
0925 04:42:51.717846 (+ 1393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:51.717850 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:51.717902 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:51.718013 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:42:51.718976 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:51.719513 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:51.720770 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:51.721051 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:51.721146 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:51.722333 (+ 1187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:42:51.722337 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:42:51.722339 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:42:51.722354 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:42:51.722377 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:42:51.722387 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:42:51.722453 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:51.722720 (+ 267us) server_negotiation.cc:299] Negotiation successful
0925 04:42:51.722791 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 04:42:51.723948 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:51.715490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37235 (local address 127.0.98.3:48323)
0925 04:42:51.716292 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:51.716348 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 04:42:51.717655 (+ 1307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:42:51.718174 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:42:51.718183 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:42:51.718278 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:42:51.718819 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:51.718833 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:51.719666 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:51.719670 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:51.720632 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:51.720643 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:51.721232 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:51.721235 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:51.721396 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:51.722022 (+ 626us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:42:51.722024 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:42:51.722073 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:42:51.722155 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:42:51.722568 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:42:51.722577 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:42:51.722583 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 04:42:51.722706 (+ 123us) client_negotiation.cc:240] Negotiation successful
0925 04:42:51.722996 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":138,"threads_started":1}
I20260925 04:42:51.725200 18429 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37235
I20260925 04:42:51.725425 18429 heartbeater.cc:464] Registering TS with master...
I20260925 04:42:51.725958 18429 heartbeater.cc:511] Master 127.0.98.62:37235 requested a full tablet report, sending...
I20260925 04:42:51.727041 17979 ts_manager.cc:194] Registered new tserver with Master: bcf0806bee8b44ecbf5528701baaa063 (127.0.98.3:33311)
I20260925 04:42:51.727690 17979 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:48323
I20260925 04:42:51.729741 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:42:51.735355 18018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:51.731706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49692 (local address 127.0.98.62:37235)
0925 04:42:51.731839 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:51.731844 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:51.731995 (+ 151us) server_negotiation.cc:407] Connection header received
0925 04:42:51.732106 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:51.732108 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:51.732152 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:51.732213 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:42:51.732472 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:51.733017 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:51.733863 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:51.734125 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:51.734216 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:51.734976 (+ 760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:42:51.734979 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:42:51.734982 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:42:51.734995 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:42:51.735014 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:42:51.735022 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:42:51.735081 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:51.735141 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 04:42:51.735194 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 04:42:51.738620 17979 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49692:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:42:51.741953 17979 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:42:51.756953 18441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:51.751185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41741 (local address 127.0.0.1:57684)
0925 04:42:51.752061 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:51.752074 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:42:51.752160 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:42:51.752743 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:42:51.752749 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:42:51.752771 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:42:51.753012 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:51.753020 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:51.754123 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:51.754126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:51.755331 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:51.755355 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:51.756363 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:51.756366 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:51.756481 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:51.756523 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 04:42:51.756642 (+ 119us) client_negotiation.cc:240] Negotiation successful
0925 04:42:51.756783 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":816,"mutex_wait_us":2,"thread_start_us":125,"threads_started":1}
I20260925 04:42:51.756986 18443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:51.751503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57684 (local address 127.0.98.2:41741)
0925 04:42:51.751930 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:51.751942 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:51.752252 (+ 310us) server_negotiation.cc:407] Connection header received
0925 04:42:51.752335 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:51.752341 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:51.752575 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:51.752737 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:42:51.753148 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:51.754005 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:51.755453 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:51.756214 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:51.756329 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:51.756549 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:51.756712 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 04:42:51.756826 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":124,"threads_started":1}
I20260925 04:42:51.757161 18444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:51.751876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45405 (local address 127.0.0.1:47494)
0925 04:42:51.752226 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:51.752237 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 04:42:51.752311 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:42:51.753010 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:42:51.753013 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:42:51.753031 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:42:51.753202 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:51.753210 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:51.754713 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:51.754717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:51.755754 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:51.755763 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:51.756772 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:51.756776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:51.756863 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:51.756898 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 04:42:51.756947 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 04:42:51.757018 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"mutex_wait_us":46,"thread_start_us":82,"threads_started":1}
I20260925 04:42:51.757802 18445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:51.752271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47494 (local address 127.0.98.1:45405)
0925 04:42:51.752577 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:51.752586 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:51.752606 (+ 20us) server_negotiation.cc:407] Connection header received
0925 04:42:51.752667 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:51.752672 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:51.752855 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:51.753013 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:42:51.753342 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:51.754578 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:51.755868 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:51.756641 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:51.757273 (+ 632us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:51.757479 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:51.757538 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 04:42:51.757640 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":112,"threads_started":1}
I20260925 04:42:51.758011 18440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:51.751116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33311 (local address 127.0.0.1:43384)
0925 04:42:51.751819 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:51.751873 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 04:42:51.752002 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:42:51.753182 (+ 1180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:42:51.753185 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:42:51.753204 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:42:51.753287 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:51.753293 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:51.754713 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:51.754717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:51.756548 (+ 1831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:51.756555 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:51.757634 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:51.757638 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:51.757727 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:51.757750 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 04:42:51.757810 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 04:42:51.757888 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"mutex_wait_us":959,"thread_start_us":100,"threads_started":1}
I20260925 04:42:51.758308 18442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:51.751245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43384 (local address 127.0.98.3:33311)
0925 04:42:51.752660 (+ 1415us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:51.752672 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:51.752695 (+ 23us) server_negotiation.cc:407] Connection header received
0925 04:42:51.752764 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:51.752768 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:51.752935 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:51.753221 (+ 286us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:42:51.753396 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:51.754593 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:51.756671 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:51.757467 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:51.757614 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:51.757790 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:51.757843 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 04:42:51.758152 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1308,"thread_start_us":159,"threads_started":1}
I20260925 04:42:51.760792 18101 tablet_service.cc:1467] Processing CreateTablet for tablet f131b23c85fd40fdab04674f16da31fe (DEFAULT_TABLE table=test-table [id=4415b15950074f67a16c5ccfcda204d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:42:51.760855 18232 tablet_service.cc:1467] Processing CreateTablet for tablet f131b23c85fd40fdab04674f16da31fe (DEFAULT_TABLE table=test-table [id=4415b15950074f67a16c5ccfcda204d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:42:51.761868 18101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f131b23c85fd40fdab04674f16da31fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:42:51.761931 18232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f131b23c85fd40fdab04674f16da31fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:42:51.761686 18363 tablet_service.cc:1467] Processing CreateTablet for tablet f131b23c85fd40fdab04674f16da31fe (DEFAULT_TABLE table=test-table [id=4415b15950074f67a16c5ccfcda204d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:42:51.762715 18363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f131b23c85fd40fdab04674f16da31fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:42:51.768661 18446 tablet_bootstrap.cc:492] T f131b23c85fd40fdab04674f16da31fe P 1e580a9e4e5f4d74878558b8c31843b0: Bootstrap starting.
I20260925 04:42:51.770159 18448 tablet_bootstrap.cc:492] T f131b23c85fd40fdab04674f16da31fe P bcf0806bee8b44ecbf5528701baaa063: Bootstrap starting.
I20260925 04:42:51.770234 18447 tablet_bootstrap.cc:492] T f131b23c85fd40fdab04674f16da31fe P de6310719e08466594f493248403cc38: Bootstrap starting.
I20260925 04:42:51.771068 18446 tablet_bootstrap.cc:654] T f131b23c85fd40fdab04674f16da31fe P 1e580a9e4e5f4d74878558b8c31843b0: Neither blocks nor log segments found. Creating new log.
I20260925 04:42:51.771914 18446 log.cc:824] T f131b23c85fd40fdab04674f16da31fe P 1e580a9e4e5f4d74878558b8c31843b0: Log is configured to *not* fsync() on all Append() calls
I20260925 04:42:51.772465 18447 tablet_bootstrap.cc:654] T f131b23c85fd40fdab04674f16da31fe P de6310719e08466594f493248403cc38: Neither blocks nor log segments found. Creating new log.
I20260925 04:42:51.772464 18448 tablet_bootstrap.cc:654] T f131b23c85fd40fdab04674f16da31fe P bcf0806bee8b44ecbf5528701baaa063: Neither blocks nor log segments found. Creating new log.
I20260925 04:42:51.773342 18448 log.cc:824] T f131b23c85fd40fdab04674f16da31fe P bcf0806bee8b44ecbf5528701baaa063: Log is configured to *not* fsync() on all Append() calls
I20260925 04:42:51.773348 18447 log.cc:824] T f131b23c85fd40fdab04674f16da31fe P de6310719e08466594f493248403cc38: Log is configured to *not* fsync() on all Append() calls
I20260925 04:42:51.774631 18446 tablet_bootstrap.cc:492] T f131b23c85fd40fdab04674f16da31fe P 1e580a9e4e5f4d74878558b8c31843b0: No bootstrap required, opened a new log
I20260925 04:42:51.774857 18446 ts_tablet_manager.cc:1397] T f131b23c85fd40fdab04674f16da31fe P 1e580a9e4e5f4d74878558b8c31843b0: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 04:42:51.775266 18448 tablet_bootstrap.cc:492] T f131b23c85fd40fdab04674f16da31fe P bcf0806bee8b44ecbf5528701baaa063: No bootstrap required, opened a new log
I20260925 04:42:51.775410 18447 tablet_bootstrap.cc:492] T f131b23c85fd40fdab04674f16da31fe P de6310719e08466594f493248403cc38: No bootstrap required, opened a new log
I20260925 04:42:51.775446 18448 ts_tablet_manager.cc:1397] T f131b23c85fd40fdab04674f16da31fe P bcf0806bee8b44ecbf5528701baaa063: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 04:42:51.775629 18447 ts_tablet_manager.cc:1397] T f131b23c85fd40fdab04674f16da31fe P de6310719e08466594f493248403cc38: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 04:42:51.778748 18448 raft_consensus.cc:348] T f131b23c85fd40fdab04674f16da31fe P bcf0806bee8b44ecbf5528701baaa063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcf0806bee8b44ecbf5528701baaa063" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33311 } } peers { permanent_uuid: "de6310719e08466594f493248403cc38" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41741 } } peers { permanent_uuid: "1e580a9e4e5f4d74878558b8c31843b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45405 } }
I20260925 04:42:51.779013 18448 raft_consensus.cc:374] T f131b23c85fd40fdab04674f16da31fe P bcf0806bee8b44ecbf5528701baaa063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:42:51.779066 18448 raft_consensus.cc:729] T f131b23c85fd40fdab04674f16da31fe P bcf0806bee8b44ecbf5528701baaa063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf0806bee8b44ecbf5528701baaa063, State: Initialized, Role: FOLLOWER
I20260925 04:42:51.779543 18448 consensus_queue.cc:260] T f131b23c85fd40fdab04674f16da31fe P bcf0806bee8b44ecbf5528701baaa063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcf0806bee8b44ecbf5528701baaa063" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33311 } } peers { permanent_uuid: "de6310719e08466594f493248403cc38" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41741 } } peers { permanent_uuid: "1e580a9e4e5f4d74878558b8c31843b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45405 } }
I20260925 04:42:51.779501 18446 raft_consensus.cc:348] T f131b23c85fd40fdab04674f16da31fe P 1e580a9e4e5f4d74878558b8c31843b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcf0806bee8b44ecbf5528701baaa063" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33311 } } peers { permanent_uuid: "de6310719e08466594f493248403cc38" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41741 } } peers { permanent_uuid: "1e580a9e4e5f4d74878558b8c31843b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45405 } }
I20260925 04:42:51.779808 18446 raft_consensus.cc:374] T f131b23c85fd40fdab04674f16da31fe P 1e580a9e4e5f4d74878558b8c31843b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:42:51.779896 18446 raft_consensus.cc:729] T f131b23c85fd40fdab04674f16da31fe P 1e580a9e4e5f4d74878558b8c31843b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e580a9e4e5f4d74878558b8c31843b0, State: Initialized, Role: FOLLOWER
I20260925 04:42:51.779979 18447 raft_consensus.cc:348] T f131b23c85fd40fdab04674f16da31fe P de6310719e08466594f493248403cc38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcf0806bee8b44ecbf5528701baaa063" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33311 } } peers { permanent_uuid: "de6310719e08466594f493248403cc38" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41741 } } peers { permanent_uuid: "1e580a9e4e5f4d74878558b8c31843b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45405 } }
I20260925 04:42:51.780223 18447 raft_consensus.cc:374] T f131b23c85fd40fdab04674f16da31fe P de6310719e08466594f493248403cc38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:42:51.780267 18447 raft_consensus.cc:729] T f131b23c85fd40fdab04674f16da31fe P de6310719e08466594f493248403cc38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de6310719e08466594f493248403cc38, State: Initialized, Role: FOLLOWER
I20260925 04:42:51.780354 18446 consensus_queue.cc:260] T f131b23c85fd40fdab04674f16da31fe P 1e580a9e4e5f4d74878558b8c31843b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcf0806bee8b44ecbf5528701baaa063" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33311 } } peers { permanent_uuid: "de6310719e08466594f493248403cc38" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41741 } } peers { permanent_uuid: "1e580a9e4e5f4d74878558b8c31843b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45405 } }
I20260925 04:42:51.780684 18447 consensus_queue.cc:260] T f131b23c85fd40fdab04674f16da31fe P de6310719e08466594f493248403cc38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcf0806bee8b44ecbf5528701baaa063" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33311 } } peers { permanent_uuid: "de6310719e08466594f493248403cc38" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41741 } } peers { permanent_uuid: "1e580a9e4e5f4d74878558b8c31843b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45405 } }
I20260925 04:42:51.780887 18448 ts_tablet_manager.cc:1428] T f131b23c85fd40fdab04674f16da31fe P bcf0806bee8b44ecbf5528701baaa063: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 04:42:51.781262 18429 heartbeater.cc:503] Master 127.0.98.62:37235 was elected leader, sending a full tablet report...
I20260925 04:42:51.781497 18446 ts_tablet_manager.cc:1428] T f131b23c85fd40fdab04674f16da31fe P 1e580a9e4e5f4d74878558b8c31843b0: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260925 04:42:51.781479 18447 ts_tablet_manager.cc:1428] T f131b23c85fd40fdab04674f16da31fe P de6310719e08466594f493248403cc38: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260925 04:42:51.781838 18167 heartbeater.cc:503] Master 127.0.98.62:37235 was elected leader, sending a full tablet report...
I20260925 04:42:51.781876 18298 heartbeater.cc:503] Master 127.0.98.62:37235 was elected leader, sending a full tablet report...
W20260925 04:42:51.812135 18168 tablet.cc:2369] T f131b23c85fd40fdab04674f16da31fe P 1e580a9e4e5f4d74878558b8c31843b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:42:51.823701 18454 raft_consensus.cc:482] T f131b23c85fd40fdab04674f16da31fe P de6310719e08466594f493248403cc38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:42:51.824087 18454 raft_consensus.cc:504] T f131b23c85fd40fdab04674f16da31fe P de6310719e08466594f493248403cc38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcf0806bee8b44ecbf5528701baaa063" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33311 } } peers { permanent_uuid: "de6310719e08466594f493248403cc38" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41741 } } peers { permanent_uuid: "1e580a9e4e5f4d74878558b8c31843b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45405 } }
I20260925 04:42:51.825392 18454 leader_election.cc:288] T f131b23c85fd40fdab04674f16da31fe P de6310719e08466594f493248403cc38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcf0806bee8b44ecbf5528701baaa063 (127.0.98.3:33311), 1e580a9e4e5f4d74878558b8c31843b0 (127.0.98.1:45405)
I20260925 04:42:51.829144 18300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:51.825688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33311 (local address 127.0.98.2:58647)
0925 04:42:51.825824 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:51.825840 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:42:51.825946 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:42:51.826224 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:42:51.826228 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:42:51.826248 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:42:51.826379 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:51.826387 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:51.827119 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:51.827122 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:51.828023 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:51.828032 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:51.828696 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:51.828699 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:51.828812 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:51.828834 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 04:42:51.828920 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 04:42:51.828995 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 04:42:51.829183 18442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:51.825816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58647 (local address 127.0.98.3:33311)
0925 04:42:51.825937 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:51.825941 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:51.825956 (+ 15us) server_negotiation.cc:407] Connection header received
0925 04:42:51.826048 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:51.826051 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:51.826101 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:51.826183 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:42:51.826484 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:51.827013 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:51.828145 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:51.828580 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:51.828678 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:51.828747 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:51.828948 (+ 201us) server_negotiation.cc:299] Negotiation successful
0925 04:42:51.829033 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 04:42:51.829913 18445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:51.826191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57467 (local address 127.0.98.1:45405)
0925 04:42:51.826387 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:51.826391 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:51.826524 (+ 133us) server_negotiation.cc:407] Connection header received
0925 04:42:51.826632 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:51.826635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:51.826679 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:51.826742 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:42:51.827009 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:51.827490 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:51.828546 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:51.829054 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:51.829152 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:51.829210 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:51.829706 (+ 496us) server_negotiation.cc:299] Negotiation successful
0925 04:42:51.829792 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260925 04:42:51.830185 18383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f131b23c85fd40fdab04674f16da31fe" candidate_uuid: "de6310719e08466594f493248403cc38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf0806bee8b44ecbf5528701baaa063" is_pre_election: true
I20260925 04:42:51.830350 18455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:51.826054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45405 (local address 127.0.98.2:57467)
0925 04:42:51.826429 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:51.826440 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 04:42:51.826537 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:42:51.826763 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:42:51.826766 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:42:51.826788 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:42:51.826904 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:51.826909 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:51.827633 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:51.827637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:51.828421 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:51.828430 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:51.829366 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:51.829370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:51.829537 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:51.829564 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 04:42:51.830132 (+ 568us) client_negotiation.cc:240] Negotiation successful
0925 04:42:51.830212 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":120,"threads_started":1}
I20260925 04:42:51.830557 18383 raft_consensus.cc:2454] T f131b23c85fd40fdab04674f16da31fe P bcf0806bee8b44ecbf5528701baaa063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de6310719e08466594f493248403cc38 in term 0.
I20260925 04:42:51.830937 18121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f131b23c85fd40fdab04674f16da31fe" candidate_uuid: "de6310719e08466594f493248403cc38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e580a9e4e5f4d74878558b8c31843b0" is_pre_election: true
I20260925 04:42:51.831225 18121 raft_consensus.cc:2454] T f131b23c85fd40fdab04674f16da31fe P 1e580a9e4e5f4d74878558b8c31843b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de6310719e08466594f493248403cc38 in term 0.
I20260925 04:42:51.831241 18185 leader_election.cc:302] T f131b23c85fd40fdab04674f16da31fe P de6310719e08466594f493248403cc38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bcf0806bee8b44ecbf5528701baaa063, de6310719e08466594f493248403cc38; no voters:
I20260925 04:42:51.831593 18454 raft_consensus.cc:2781] T f131b23c85fd40fdab04674f16da31fe P de6310719e08466594f493248403cc38 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:42:51.831715 18454 raft_consensus.cc:482] T f131b23c85fd40fdab04674f16da31fe P de6310719e08466594f493248403cc38 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:42:51.831768 18454 raft_consensus.cc:3037] T f131b23c85fd40fdab04674f16da31fe P de6310719e08466594f493248403cc38 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:42:51.832803 18454 raft_consensus.cc:504] T f131b23c85fd40fdab04674f16da31fe P de6310719e08466594f493248403cc38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcf0806bee8b44ecbf5528701baaa063" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33311 } } peers { permanent_uuid: "de6310719e08466594f493248403cc38" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41741 } } peers { permanent_uuid: "1e580a9e4e5f4d74878558b8c31843b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45405 } }
I20260925 04:42:51.833144 18454 leader_election.cc:288] T f131b23c85fd40fdab04674f16da31fe P de6310719e08466594f493248403cc38 [CANDIDATE]: Term 1 election: Requested vote from peers bcf0806bee8b44ecbf5528701baaa063 (127.0.98.3:33311), 1e580a9e4e5f4d74878558b8c31843b0 (127.0.98.1:45405)
I20260925 04:42:51.833627 18383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f131b23c85fd40fdab04674f16da31fe" candidate_uuid: "de6310719e08466594f493248403cc38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf0806bee8b44ecbf5528701baaa063"
I20260925 04:42:51.833648 18121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f131b23c85fd40fdab04674f16da31fe" candidate_uuid: "de6310719e08466594f493248403cc38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e580a9e4e5f4d74878558b8c31843b0"
I20260925 04:42:51.833755 18383 raft_consensus.cc:3037] T f131b23c85fd40fdab04674f16da31fe P bcf0806bee8b44ecbf5528701baaa063 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:42:51.833844 18121 raft_consensus.cc:3037] T f131b23c85fd40fdab04674f16da31fe P 1e580a9e4e5f4d74878558b8c31843b0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:42:51.834857 18383 raft_consensus.cc:2454] T f131b23c85fd40fdab04674f16da31fe P bcf0806bee8b44ecbf5528701baaa063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de6310719e08466594f493248403cc38 in term 1.
I20260925 04:42:51.834913 18121 raft_consensus.cc:2454] T f131b23c85fd40fdab04674f16da31fe P 1e580a9e4e5f4d74878558b8c31843b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de6310719e08466594f493248403cc38 in term 1.
I20260925 04:42:51.835212 18185 leader_election.cc:302] T f131b23c85fd40fdab04674f16da31fe P de6310719e08466594f493248403cc38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bcf0806bee8b44ecbf5528701baaa063, de6310719e08466594f493248403cc38; no voters:
I20260925 04:42:51.835389 18454 raft_consensus.cc:2781] T f131b23c85fd40fdab04674f16da31fe P de6310719e08466594f493248403cc38 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:42:51.835641 18454 raft_consensus.cc:686] T f131b23c85fd40fdab04674f16da31fe P de6310719e08466594f493248403cc38 [term 1 LEADER]: Becoming Leader. State: Replica: de6310719e08466594f493248403cc38, State: Running, Role: LEADER
I20260925 04:42:51.836040 18454 consensus_queue.cc:237] T f131b23c85fd40fdab04674f16da31fe P de6310719e08466594f493248403cc38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcf0806bee8b44ecbf5528701baaa063" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33311 } } peers { permanent_uuid: "de6310719e08466594f493248403cc38" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41741 } } peers { permanent_uuid: "1e580a9e4e5f4d74878558b8c31843b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45405 } }
I20260925 04:42:51.839484 17979 catalog_manager.cc:5495] T f131b23c85fd40fdab04674f16da31fe P de6310719e08466594f493248403cc38 reported cstate change: term changed from 0 to 1, leader changed from <none> to de6310719e08466594f493248403cc38 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "de6310719e08466594f493248403cc38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcf0806bee8b44ecbf5528701baaa063" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de6310719e08466594f493248403cc38" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41741 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e580a9e4e5f4d74878558b8c31843b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45405 } health_report { overall_health: UNKNOWN } } }
I20260925 04:42:51.857559 18443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:51.853814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57700 (local address 127.0.98.2:41741)
0925 04:42:51.853968 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:51.853973 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:51.853988 (+ 15us) server_negotiation.cc:407] Connection header received
0925 04:42:51.854134 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:51.854136 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:51.854194 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:51.854282 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:42:51.854526 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:51.855017 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:51.855993 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:51.856202 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:51.856289 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:51.856629 (+ 340us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:42:51.857163 (+ 534us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:42:51.857279 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:51.857350 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 04:42:51.857416 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 04:42:51.869457 18383 raft_consensus.cc:1261] T f131b23c85fd40fdab04674f16da31fe P bcf0806bee8b44ecbf5528701baaa063 [term 1 FOLLOWER]: Refusing update from remote peer de6310719e08466594f493248403cc38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:42:51.869726 18121 raft_consensus.cc:1261] T f131b23c85fd40fdab04674f16da31fe P 1e580a9e4e5f4d74878558b8c31843b0 [term 1 FOLLOWER]: Refusing update from remote peer de6310719e08466594f493248403cc38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:42:51.870405 18454 consensus_queue.cc:1035] T f131b23c85fd40fdab04674f16da31fe P de6310719e08466594f493248403cc38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e580a9e4e5f4d74878558b8c31843b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:42:51.870685 18456 consensus_queue.cc:1035] T f131b23c85fd40fdab04674f16da31fe P de6310719e08466594f493248403cc38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf0806bee8b44ecbf5528701baaa063" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:42:51.882009 18460 mvcc.cc:205] Tried to move back new op lower bound from 7333115379151532032 to 7333115379043938304. Current Snapshot: MvccSnapshot[applied={T|T < 7333115379151532032}]
I20260925 04:42:51.883074 18461 mvcc.cc:205] Tried to move back new op lower bound from 7333115379151532032 to 7333115379043938304. Current Snapshot: MvccSnapshot[applied={T|T < 7333115379151532032 or (T in {7333115379151532032})}]
I20260925 04:42:51.888357 17978 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:42:51.888576 17978 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:42:51.891291 17978 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5e745bfc95c94f0694b43e01d761e52a: Sending DeleteTablet for 3 replicas of tablet f131b23c85fd40fdab04674f16da31fe
I20260925 04:42:51.892233 18232 tablet_service.cc:1514] Processing DeleteTablet for tablet f131b23c85fd40fdab04674f16da31fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:42:51 UTC) from {username='slave'} at 127.0.0.1:57684
I20260925 04:42:51.892308 18363 tablet_service.cc:1514] Processing DeleteTablet for tablet f131b23c85fd40fdab04674f16da31fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:42:51 UTC) from {username='slave'} at 127.0.0.1:43384
I20260925 04:42:51.892791 18469 tablet_replica.cc:330] T f131b23c85fd40fdab04674f16da31fe P de6310719e08466594f493248403cc38: stopping tablet replica
I20260925 04:42:51.893180 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 18039
I20260925 04:42:51.893147 18101 tablet_service.cc:1514] Processing DeleteTablet for tablet f131b23c85fd40fdab04674f16da31fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:42:51 UTC) from {username='slave'} at 127.0.0.1:47494
I20260925 04:42:51.893210 18469 raft_consensus.cc:2229] T f131b23c85fd40fdab04674f16da31fe P de6310719e08466594f493248403cc38 [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:42:51.894438 18469 raft_consensus.cc:2258] T f131b23c85fd40fdab04674f16da31fe P de6310719e08466594f493248403cc38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:42:51.894564 18470 tablet_replica.cc:330] T f131b23c85fd40fdab04674f16da31fe P bcf0806bee8b44ecbf5528701baaa063: stopping tablet replica
I20260925 04:42:51.895108 18470 raft_consensus.cc:2229] T f131b23c85fd40fdab04674f16da31fe P bcf0806bee8b44ecbf5528701baaa063 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:42:51.895473 18470 raft_consensus.cc:2258] T f131b23c85fd40fdab04674f16da31fe P bcf0806bee8b44ecbf5528701baaa063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:42:51.896536 18469 ts_tablet_manager.cc:1905] T f131b23c85fd40fdab04674f16da31fe P de6310719e08466594f493248403cc38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:42:51.898087 18470 ts_tablet_manager.cc:1905] T f131b23c85fd40fdab04674f16da31fe P bcf0806bee8b44ecbf5528701baaa063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:42:51.900763 18469 ts_tablet_manager.cc:1918] T f131b23c85fd40fdab04674f16da31fe P de6310719e08466594f493248403cc38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:42:51.901026 18469 log.cc:1196] T f131b23c85fd40fdab04674f16da31fe P de6310719e08466594f493248403cc38: Deleting WAL directory at /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311123478858-392-0/minicluster-data/ts-1/wal/wals/f131b23c85fd40fdab04674f16da31fe
I20260925 04:42:51.901196 18470 ts_tablet_manager.cc:1918] T f131b23c85fd40fdab04674f16da31fe P bcf0806bee8b44ecbf5528701baaa063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:42:51.901319 18470 log.cc:1196] T f131b23c85fd40fdab04674f16da31fe P bcf0806bee8b44ecbf5528701baaa063: Deleting WAL directory at /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311123478858-392-0/minicluster-data/ts-2/wal/wals/f131b23c85fd40fdab04674f16da31fe
I20260925 04:42:51.901670 18470 ts_tablet_manager.cc:1939] T f131b23c85fd40fdab04674f16da31fe P bcf0806bee8b44ecbf5528701baaa063: Deleting consensus metadata
I20260925 04:42:51.901670 18469 ts_tablet_manager.cc:1939] T f131b23c85fd40fdab04674f16da31fe P de6310719e08466594f493248403cc38: Deleting consensus metadata
W20260925 04:42:51.902872 17963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45405 (error 108)
I20260925 04:42:51.903425 17963 catalog_manager.cc:4841] TS bcf0806bee8b44ecbf5528701baaa063 (127.0.98.3:33311): tablet f131b23c85fd40fdab04674f16da31fe (table test-table [id=4415b15950074f67a16c5ccfcda204d4]) successfully deleted
I20260925 04:42:51.903820 17966 catalog_manager.cc:4841] TS de6310719e08466594f493248403cc38 (127.0.98.2:41741): tablet f131b23c85fd40fdab04674f16da31fe (table test-table [id=4415b15950074f67a16c5ccfcda204d4]) successfully deleted
W20260925 04:42:51.904109 18441 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:42:51.903785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45405 (local address 127.0.0.1:47508)
0925 04:42:51.903926 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:51.904005 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45405: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 04:42:51.904194 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 18170
W20260925 04:42:51.904266 17963 catalog_manager.cc:4584] TS 1e580a9e4e5f4d74878558b8c31843b0 (127.0.98.1:45405): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f131b23c85fd40fdab04674f16da31fe: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45405: connect: Connection reset by peer (error 104)
I20260925 04:42:51.909503 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 18301
I20260925 04:42:51.915241 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 17948
2026-09-25T04:42:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1069 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:42:51 dist-test-slave-dc95 krb5kdc[18474](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:42:51 dist-test-slave-dc95 krb5kdc[18474](info): set up 2 sockets
Sep 25 04:42:51 dist-test-slave-dc95 krb5kdc[18474](info): commencing operation
krb5kdc: starting...
W20260925 04:42:52.887354 392 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/krb5kdc/test-user.keytab.
Sep 25 04:42:52 dist-test-slave-dc95 krb5kdc[18474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:42:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:42:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 04:42:53.015224 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39081
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39081
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:42:53.129107 18487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:42:53.129415 18487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:42:53.129457 18487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:42:53.133960 18487 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:42:53.134058 18487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:42:53.134078 18487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:42:53.134096 18487 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:42:53.134111 18487 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:42:53.140040 18487 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39081
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39081
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:42:53.141208 18487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:42:53.142289 18487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:42:53.151355 18492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:42:53.151374 18493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:42:53.151374 18495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:42:53.151646 18487 server_base.cc:1034] running on GCE node
I20260925 04:42:53.152263 18487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:42:53.153493 18487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:42:53.154711 18487 hybrid_clock.cc:648] HybridClock initialized: now 1790311373154682 us; error 47 us; skew 500 ppm
Sep 25 04:42:53 dist-test-slave-dc95 krb5kdc[18474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311373, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:42:53.157697 18487 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 04:42:53.158680 18487 webserver.cc:460] Webserver started at http://127.0.98.62:44037/ using document root <none> and password file <none>
I20260925 04:42:53.159224 18487 fs_manager.cc:359] Metadata directory not provided
I20260925 04:42:53.159283 18487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:42:53.159453 18487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:42:53.161688 18487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/minicluster-data/master-0/data/instance:
uuid: "59e3272de5f440f7b62fe2418ea80e1b"
format_stamp: "Formatted at 2026-09-25 04:42:53 on dist-test-slave-dc95"
server_key: "877f3240b3a4442d73253a38a1e183a2"
server_key_iv: "b740dfc18aeaf3560958eb06e9029423"
server_key_version: "encryptionkey@0"
I20260925 04:42:53.162261 18487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/minicluster-data/master-0/wal/instance:
uuid: "59e3272de5f440f7b62fe2418ea80e1b"
format_stamp: "Formatted at 2026-09-25 04:42:53 on dist-test-slave-dc95"
server_key: "877f3240b3a4442d73253a38a1e183a2"
server_key_iv: "b740dfc18aeaf3560958eb06e9029423"
server_key_version: "encryptionkey@0"
I20260925 04:42:53.166275 18487 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 04:42:53.168936 18502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:53.170473 18487 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 04:42:53.170619 18487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/minicluster-data/master-0/wal
uuid: "59e3272de5f440f7b62fe2418ea80e1b"
format_stamp: "Formatted at 2026-09-25 04:42:53 on dist-test-slave-dc95"
server_key: "877f3240b3a4442d73253a38a1e183a2"
server_key_iv: "b740dfc18aeaf3560958eb06e9029423"
server_key_version: "encryptionkey@0"
I20260925 04:42:53.170841 18487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:42:53.193671 18487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:42:53.197000 18487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:42:53.197182 18487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:42:53.204869 18487 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39081
I20260925 04:42:53.204936 18554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39081 every 8 connection(s)
I20260925 04:42:53.206007 18487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
I20260925 04:42:53.208945 18555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:42:53.212323 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 18487
I20260925 04:42:53.212433 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/minicluster-data/master-0/wal/instance
I20260925 04:42:53.212719 392 external_mini_cluster.cc:1428] Setting key ad55186a998e6e07590f10128bcba988
I20260925 04:42:53.215103 18555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59e3272de5f440f7b62fe2418ea80e1b: Bootstrap starting.
I20260925 04:42:53.217669 18555 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59e3272de5f440f7b62fe2418ea80e1b: Neither blocks nor log segments found. Creating new log.
I20260925 04:42:53.218550 18555 log.cc:824] T 00000000000000000000000000000000 P 59e3272de5f440f7b62fe2418ea80e1b: Log is configured to *not* fsync() on all Append() calls
I20260925 04:42:53.221066 18555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59e3272de5f440f7b62fe2418ea80e1b: No bootstrap required, opened a new log
Sep 25 04:42:53 dist-test-slave-dc95 krb5kdc[18474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:42:53.224504 18555 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59e3272de5f440f7b62fe2418ea80e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59e3272de5f440f7b62fe2418ea80e1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39081 } }
I20260925 04:42:53.224758 18555 raft_consensus.cc:374] T 00000000000000000000000000000000 P 59e3272de5f440f7b62fe2418ea80e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:42:53.224817 18555 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59e3272de5f440f7b62fe2418ea80e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59e3272de5f440f7b62fe2418ea80e1b, State: Initialized, Role: FOLLOWER
I20260925 04:42:53.225291 18555 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59e3272de5f440f7b62fe2418ea80e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59e3272de5f440f7b62fe2418ea80e1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39081 } }
I20260925 04:42:53.225474 18555 raft_consensus.cc:388] T 00000000000000000000000000000000 P 59e3272de5f440f7b62fe2418ea80e1b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:42:53.225554 18555 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59e3272de5f440f7b62fe2418ea80e1b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:42:53.225653 18555 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59e3272de5f440f7b62fe2418ea80e1b [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:42:53.226684 18555 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59e3272de5f440f7b62fe2418ea80e1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59e3272de5f440f7b62fe2418ea80e1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39081 } }
I20260925 04:42:53.227098 18555 leader_election.cc:302] T 00000000000000000000000000000000 P 59e3272de5f440f7b62fe2418ea80e1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59e3272de5f440f7b62fe2418ea80e1b; no voters:
I20260925 04:42:53.227458 18555 leader_election.cc:288] T 00000000000000000000000000000000 P 59e3272de5f440f7b62fe2418ea80e1b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:42:53.227568 18560 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59e3272de5f440f7b62fe2418ea80e1b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:42:53.227856 18560 raft_consensus.cc:686] T 00000000000000000000000000000000 P 59e3272de5f440f7b62fe2418ea80e1b [term 1 LEADER]: Becoming Leader. State: Replica: 59e3272de5f440f7b62fe2418ea80e1b, State: Running, Role: LEADER
I20260925 04:42:53.228168 18560 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59e3272de5f440f7b62fe2418ea80e1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59e3272de5f440f7b62fe2418ea80e1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39081 } }
I20260925 04:42:53.228473 18558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:53.214110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58582 (local address 127.0.98.62:39081)
0925 04:42:53.214547 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:53.214563 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:53.214599 (+ 36us) server_negotiation.cc:407] Connection header received
0925 04:42:53.215247 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:53.215270 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:53.215673 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:53.215982 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:42:53.216852 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:53.217905 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:53.219028 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:53.219695 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:53.219821 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:53.222684 (+ 2863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:42:53.222690 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:42:53.222706 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:42:53.222736 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:42:53.225032 (+ 2296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:42:53.225590 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:42:53.225597 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:42:53.225604 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:42:53.225789 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:42:53.226089 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:42:53.226091 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:42:53.226092 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:42:53.226513 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:42:53.226664 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:53.227028 (+ 364us) server_negotiation.cc:299] Negotiation successful
0925 04:42:53.227399 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":130,"threads_started":1}
I20260925 04:42:53.228713 18555 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59e3272de5f440f7b62fe2418ea80e1b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:42:53.229446 18560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59e3272de5f440f7b62fe2418ea80e1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59e3272de5f440f7b62fe2418ea80e1b. Latest consensus state: current_term: 1 leader_uuid: "59e3272de5f440f7b62fe2418ea80e1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59e3272de5f440f7b62fe2418ea80e1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39081 } } }
I20260925 04:42:53.229560 18560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59e3272de5f440f7b62fe2418ea80e1b [sys.catalog]: This master's current role is: LEADER
I20260925 04:42:53.229871 18561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59e3272de5f440f7b62fe2418ea80e1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "59e3272de5f440f7b62fe2418ea80e1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59e3272de5f440f7b62fe2418ea80e1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39081 } } }
I20260925 04:42:53.229957 18561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59e3272de5f440f7b62fe2418ea80e1b [sys.catalog]: This master's current role is: LEADER
W20260925 04:42:53.233376 18575 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 59e3272de5f440f7b62fe2418ea80e1b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 04:42:53.233454 18575 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 04:42:53.233522 18569 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:42:53.234251 18569 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:42:53.239590 18569 catalog_manager.cc:1347] Generated new cluster ID: d9d2185a8763459da7b4c26f2a5c22cc
I20260925 04:42:53.239717 18569 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:42:53.266983 18569 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:42:53.268370 18569 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:42:53.274552 18569 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 59e3272de5f440f7b62fe2418ea80e1b: Generated new TSK 0
I20260925 04:42:53.275456 18569 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 04:42:53.306646 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39081
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:42:53.427944 18581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:42:53.428359 18581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:42:53.428409 18581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:42:53.432924 18581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:42:53.433017 18581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:42:53.433081 18581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 04:42:53.438613 18581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:42:53.439783 18581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:42:53.440820 18581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:42:53.450968 18589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:42:53.451148 18581 server_base.cc:1034] running on GCE node
W20260925 04:42:53.450968 18586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:42:53.450973 18587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:42:53.451714 18581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:42:53.452510 18581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:42:53.453744 18581 hybrid_clock.cc:648] HybridClock initialized: now 1790311373453693 us; error 66 us; skew 500 ppm
Sep 25 04:42:53 dist-test-slave-dc95 krb5kdc[18474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311373, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:42:53.457526 18581 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 04:42:53.458624 18581 webserver.cc:460] Webserver started at http://127.0.98.1:41941/ using document root <none> and password file <none>
I20260925 04:42:53.459240 18581 fs_manager.cc:359] Metadata directory not provided
I20260925 04:42:53.459309 18581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:42:53.459512 18581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:42:53.461766 18581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/minicluster-data/ts-0/data/instance:
uuid: "843bf034612d439caf70eb40bcc08b7c"
format_stamp: "Formatted at 2026-09-25 04:42:53 on dist-test-slave-dc95"
server_key: "7416a7d705f16d1f4097fe141a656f04"
server_key_iv: "83093537d655ee59cb348ff291db69d1"
server_key_version: "encryptionkey@0"
I20260925 04:42:53.462353 18581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/minicluster-data/ts-0/wal/instance:
uuid: "843bf034612d439caf70eb40bcc08b7c"
format_stamp: "Formatted at 2026-09-25 04:42:53 on dist-test-slave-dc95"
server_key: "7416a7d705f16d1f4097fe141a656f04"
server_key_iv: "83093537d655ee59cb348ff291db69d1"
server_key_version: "encryptionkey@0"
I20260925 04:42:53.466521 18581 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 04:42:53.469223 18596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:53.470562 18581 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 04:42:53.470693 18581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/minicluster-data/ts-0/wal
uuid: "843bf034612d439caf70eb40bcc08b7c"
format_stamp: "Formatted at 2026-09-25 04:42:53 on dist-test-slave-dc95"
server_key: "7416a7d705f16d1f4097fe141a656f04"
server_key_iv: "83093537d655ee59cb348ff291db69d1"
server_key_version: "encryptionkey@0"
I20260925 04:42:53.470829 18581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:42:53.488528 18581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:42:53.492142 18581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:42:53.492337 18581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:42:53.493040 18581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:42:53.494096 18581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:42:53.494164 18581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:53.494210 18581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:42:53.494225 18581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:53.504112 18581 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37549
I20260925 04:42:53.504148 18709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37549 every 8 connection(s)
I20260925 04:42:53.505113 18581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
I20260925 04:42:53.513804 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 18581
I20260925 04:42:53.513932 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/minicluster-data/ts-0/wal/instance
I20260925 04:42:53.514245 392 external_mini_cluster.cc:1428] Setting key 5e3c8dfd2fdb47356abdd43e304f452e
Sep 25 04:42:53 dist-test-slave-dc95 krb5kdc[18474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311373, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:42:53.522854 18558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:53.507229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52291 (local address 127.0.98.62:39081)
0925 04:42:53.507361 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:53.507365 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:53.508383 (+ 1018us) server_negotiation.cc:407] Connection header received
0925 04:42:53.509755 (+ 1372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:53.509758 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:53.509811 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:53.509895 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:42:53.511408 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:53.512043 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:53.513593 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:53.513842 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:53.513971 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:53.517640 (+ 3669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:42:53.517645 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:42:53.517648 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:42:53.517682 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:42:53.519276 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:42:53.520210 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:42:53.520218 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:42:53.520220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:42:53.520421 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:42:53.521115 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:42:53.521119 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:42:53.521121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:42:53.521372 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:42:53.521471 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:53.522439 (+ 968us) server_negotiation.cc:299] Negotiation successful
0925 04:42:53.522674 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 04:42:53.523962 18712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:53.507659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39081 (local address 127.0.98.1:52291)
0925 04:42:53.508196 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:53.508248 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 04:42:53.509481 (+ 1233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:42:53.510097 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:42:53.510106 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:42:53.510515 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:42:53.511211 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:53.511228 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:53.512204 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:53.512207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:53.513380 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:53.513394 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:53.514061 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:53.514067 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:53.514308 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:53.515338 (+ 1030us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:42:53.515360 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:42:53.517417 (+ 2057us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:42:53.519439 (+ 2022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:42:53.519447 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:42:53.519470 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:42:53.519972 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:42:53.520622 (+ 650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:42:53.520625 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:42:53.520628 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:42:53.520898 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:42:53.521510 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:42:53.521519 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:42:53.521944 (+ 425us) client_negotiation.cc:769] Sending connection context
0925 04:42:53.522218 (+ 274us) client_negotiation.cc:240] Negotiation successful
0925 04:42:53.522692 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":114,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 04:42:53.525591 18710 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39081
I20260925 04:42:53.525873 18710 heartbeater.cc:464] Registering TS with master...
I20260925 04:42:53.526539 18710 heartbeater.cc:511] Master 127.0.98.62:39081 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260925 04:42:53.528584 18519 ts_manager.cc:194] Registered new tserver with Master: 843bf034612d439caf70eb40bcc08b7c (127.0.98.1:37549)
I20260925 04:42:53.530043 18519 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52291
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 04:42:53.546931 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39081
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:42:53.659343 18715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:42:53.659665 18715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:42:53.659729 18715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:42:53.663903 18715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:42:53.663978 18715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:42:53.664042 18715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 04:42:53.669623 18715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:42:53.670737 18715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:42:53.671799 18715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:42:53.682092 18720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:42:53.682122 18715 server_base.cc:1034] running on GCE node
W20260925 04:42:53.682143 18721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:42:53.682142 18723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:42:53.682897 18715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:42:53.683684 18715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:42:53.684867 18715 hybrid_clock.cc:648] HybridClock initialized: now 1790311373684844 us; error 42 us; skew 500 ppm
Sep 25 04:42:53 dist-test-slave-dc95 krb5kdc[18474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311373, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:42:53.692637 18715 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 04:42:53.693681 18715 webserver.cc:460] Webserver started at http://127.0.98.2:41145/ using document root <none> and password file <none>
I20260925 04:42:53.694249 18715 fs_manager.cc:359] Metadata directory not provided
I20260925 04:42:53.694319 18715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:42:53.694535 18715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:42:53.696825 18715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/minicluster-data/ts-1/data/instance:
uuid: "d4ac50adb8b346aa8600758dec15e523"
format_stamp: "Formatted at 2026-09-25 04:42:53 on dist-test-slave-dc95"
server_key: "10315ec56ba0f269136c7b6a87dcd3b1"
server_key_iv: "46d51f1bc839b8464a4946b415377019"
server_key_version: "encryptionkey@0"
I20260925 04:42:53.697403 18715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d4ac50adb8b346aa8600758dec15e523"
format_stamp: "Formatted at 2026-09-25 04:42:53 on dist-test-slave-dc95"
server_key: "10315ec56ba0f269136c7b6a87dcd3b1"
server_key_iv: "46d51f1bc839b8464a4946b415377019"
server_key_version: "encryptionkey@0"
I20260925 04:42:53.701413 18715 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 04:42:53.704121 18730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:53.705466 18715 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:42:53.705626 18715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/minicluster-data/ts-1/wal
uuid: "d4ac50adb8b346aa8600758dec15e523"
format_stamp: "Formatted at 2026-09-25 04:42:53 on dist-test-slave-dc95"
server_key: "10315ec56ba0f269136c7b6a87dcd3b1"
server_key_iv: "46d51f1bc839b8464a4946b415377019"
server_key_version: "encryptionkey@0"
I20260925 04:42:53.705754 18715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:42:53.740608 18715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:42:53.744207 18715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:42:53.744408 18715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:42:53.745028 18715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:42:53.745925 18715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:42:53.745973 18715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:53.746028 18715 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:42:53.746054 18715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:53.754794 18715 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38977
I20260925 04:42:53.754825 18843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38977 every 8 connection(s)
I20260925 04:42:53.755757 18715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
I20260925 04:42:53.764238 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 18715
I20260925 04:42:53.764353 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/minicluster-data/ts-1/wal/instance
I20260925 04:42:53.764612 392 external_mini_cluster.cc:1428] Setting key 3a1b74ef418ad84339465140adf6f99b
Sep 25 04:42:53 dist-test-slave-dc95 krb5kdc[18474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311373, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:42:53.770117 18558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:53.757703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41665 (local address 127.0.98.62:39081)
0925 04:42:53.757888 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:53.757892 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:53.758835 (+ 943us) server_negotiation.cc:407] Connection header received
0925 04:42:53.760147 (+ 1312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:53.760150 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:53.760201 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:53.760276 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:42:53.761496 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:53.762070 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:53.763249 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:53.763523 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:53.763639 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:53.766260 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:42:53.766264 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:42:53.766266 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:42:53.766290 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:42:53.767721 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:42:53.768340 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:42:53.768343 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:42:53.768344 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:42:53.768418 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:42:53.768799 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:42:53.768802 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:42:53.768803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:42:53.769087 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:42:53.769185 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:53.769781 (+ 596us) server_negotiation.cc:299] Negotiation successful
0925 04:42:53.769920 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 04:42:53.771035 18846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:53.758194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39081 (local address 127.0.98.2:41665)
0925 04:42:53.758667 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:53.758711 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 04:42:53.759906 (+ 1195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:42:53.760438 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:42:53.760445 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:42:53.760838 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:42:53.761339 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:53.761352 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:53.762196 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:53.762199 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:53.763120 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:53.763129 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:53.763705 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:53.763709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:53.763857 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:53.764472 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:42:53.764491 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:42:53.766086 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:42:53.767845 (+ 1759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:42:53.767852 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:42:53.767872 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:42:53.768244 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:42:53.768505 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:42:53.768508 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:42:53.768510 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:42:53.768692 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:42:53.769212 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:42:53.769218 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:42:53.769531 (+ 313us) client_negotiation.cc:769] Sending connection context
0925 04:42:53.769710 (+ 179us) client_negotiation.cc:240] Negotiation successful
0925 04:42:53.770035 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":115,"threads_started":1}
I20260925 04:42:53.772284 18844 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39081
I20260925 04:42:53.772540 18844 heartbeater.cc:464] Registering TS with master...
I20260925 04:42:53.773065 18844 heartbeater.cc:511] Master 127.0.98.62:39081 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260925 04:42:53.774121 18519 ts_manager.cc:194] Registered new tserver with Master: d4ac50adb8b346aa8600758dec15e523 (127.0.98.2:38977)
I20260925 04:42:53.774868 18519 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41665
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 04:42:53.793226 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39081
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:42:53.907308 18849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:42:53.907642 18849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:42:53.907698 18849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:42:53.912557 18849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:42:53.912645 18849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:42:53.912710 18849 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 04:42:53.918373 18849 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18849
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:42:53.919456 18849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:42:53.920554 18849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:42:53.930369 18855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:42:53.930408 18854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:42:53.930382 18857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:42:53.930433 18849 server_base.cc:1034] running on GCE node
I20260925 04:42:53.931083 18849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:42:53.931788 18849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:42:53.932967 18849 hybrid_clock.cc:648] HybridClock initialized: now 1790311373932945 us; error 34 us; skew 500 ppm
Sep 25 04:42:53 dist-test-slave-dc95 krb5kdc[18474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311373, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:42:53.935741 18849 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 04:42:53.936648 18849 webserver.cc:460] Webserver started at http://127.0.98.3:32913/ using document root <none> and password file <none>
I20260925 04:42:53.937202 18849 fs_manager.cc:359] Metadata directory not provided
I20260925 04:42:53.937268 18849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:42:53.937454 18849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:42:53.939642 18849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/minicluster-data/ts-2/data/instance:
uuid: "b92dc1f17c7b4d858081cddeb9612506"
format_stamp: "Formatted at 2026-09-25 04:42:53 on dist-test-slave-dc95"
server_key: "a064bf4669b143f25becfcdfe84906c4"
server_key_iv: "3b50643953e4eb4ba2da784ca364a812"
server_key_version: "encryptionkey@0"
I20260925 04:42:53.940198 18849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b92dc1f17c7b4d858081cddeb9612506"
format_stamp: "Formatted at 2026-09-25 04:42:53 on dist-test-slave-dc95"
server_key: "a064bf4669b143f25becfcdfe84906c4"
server_key_iv: "3b50643953e4eb4ba2da784ca364a812"
server_key_version: "encryptionkey@0"
I20260925 04:42:53.944103 18849 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:42:53.946673 18864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:53.948050 18849 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:42:53.948163 18849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/minicluster-data/ts-2/wal
uuid: "b92dc1f17c7b4d858081cddeb9612506"
format_stamp: "Formatted at 2026-09-25 04:42:53 on dist-test-slave-dc95"
server_key: "a064bf4669b143f25becfcdfe84906c4"
server_key_iv: "3b50643953e4eb4ba2da784ca364a812"
server_key_version: "encryptionkey@0"
I20260925 04:42:53.948280 18849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:42:53.977232 18849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:42:53.980458 18849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:42:53.980638 18849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:42:53.981237 18849 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:42:53.982085 18849 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:42:53.982132 18849 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:53.982184 18849 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:42:53.982210 18849 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:53.990767 18849 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33377
I20260925 04:42:53.990790 18977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33377 every 8 connection(s)
I20260925 04:42:53.991724 18849 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
I20260925 04:42:54.000154 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 18849
I20260925 04:42:54.000279 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/minicluster-data/ts-2/wal/instance
I20260925 04:42:54.000560 392 external_mini_cluster.cc:1428] Setting key 8a4e956c439b69d871c6d6f5c2632cee
Sep 25 04:42:53 dist-test-slave-dc95 krb5kdc[18474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311373, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:42:54.005080 18558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:53.993553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50455 (local address 127.0.98.62:39081)
0925 04:42:53.993681 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:53.993685 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:53.994496 (+ 811us) server_negotiation.cc:407] Connection header received
0925 04:42:53.995699 (+ 1203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:53.995702 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:53.995750 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:53.995821 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:42:53.997019 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:53.997551 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:53.998669 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:53.998924 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:53.999006 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:54.001764 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:42:54.001767 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:42:54.001769 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:42:54.001793 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:42:54.003090 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:42:54.003657 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:42:54.003660 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:42:54.003662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:42:54.003750 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:42:54.004092 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:42:54.004095 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:42:54.004096 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:42:54.004323 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:42:54.004424 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:54.004860 (+ 436us) server_negotiation.cc:299] Negotiation successful
0925 04:42:54.004961 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 04:42:54.005898 18980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:53.993880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39081 (local address 127.0.98.3:50455)
0925 04:42:53.994334 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:53.994379 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 04:42:53.995483 (+ 1104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:42:53.996012 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:42:53.996019 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:42:53.996380 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:42:53.996873 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:53.996887 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:53.997640 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:53.997642 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:53.998553 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:53.998562 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:53.999085 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:53.999088 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:53.999246 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:53.999838 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:42:53.999855 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:42:54.001591 (+ 1736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:42:54.003216 (+ 1625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:42:54.003221 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:42:54.003236 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:42:54.003527 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:42:54.003826 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:42:54.003828 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:42:54.003830 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:42:54.003980 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:42:54.004411 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:42:54.004422 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:42:54.004668 (+ 246us) client_negotiation.cc:769] Sending connection context
0925 04:42:54.004823 (+ 155us) client_negotiation.cc:240] Negotiation successful
0925 04:42:54.005082 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":97,"threads_started":1}
I20260925 04:42:54.007042 18978 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39081
I20260925 04:42:54.007278 18978 heartbeater.cc:464] Registering TS with master...
I20260925 04:42:54.007845 18978 heartbeater.cc:511] Master 127.0.98.62:39081 requested a full tablet report, sending...
I20260925 04:42:54.008888 18519 ts_manager.cc:194] Registered new tserver with Master: b92dc1f17c7b4d858081cddeb9612506 (127.0.98.3:33377)
I20260925 04:42:54.009460 18519 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50455
I20260925 04:42:54.014312 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:42:54.024148 18558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:54.016295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58588 (local address 127.0.98.62:39081)
0925 04:42:54.016426 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:54.016431 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:54.016582 (+ 151us) server_negotiation.cc:407] Connection header received
0925 04:42:54.016677 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:54.016679 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:54.016727 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:54.016791 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:42:54.017477 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:54.018012 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:54.018825 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:54.019074 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:54.019158 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:54.020873 (+ 1715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:42:54.020877 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:42:54.020879 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:42:54.020907 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:42:54.022270 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:42:54.022816 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:42:54.022819 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:42:54.022820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:42:54.022895 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:42:54.023192 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:42:54.023195 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:42:54.023196 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:42:54.023418 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:42:54.023539 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:54.023887 (+ 348us) server_negotiation.cc:299] Negotiation successful
0925 04:42:54.023977 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 04:42:54.027487 18519 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58588:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:42:54.030812 18519 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:42:54.045929 18992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:54.040362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55066 (local address 127.0.98.2:38977)
0925 04:42:54.040817 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:54.040830 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:54.041545 (+ 715us) server_negotiation.cc:407] Connection header received
0925 04:42:54.041615 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:54.041619 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:54.041775 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:54.041956 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:42:54.042338 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:54.043092 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:54.044495 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:54.045073 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:54.045189 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:54.045372 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:54.045656 (+ 284us) server_negotiation.cc:299] Negotiation successful
0925 04:42:54.045765 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":96,"threads_started":1}
I20260925 04:42:54.045923 18989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:54.040168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38977 (local address 127.0.0.1:55066)
0925 04:42:54.041209 (+ 1041us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:54.041265 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 04:42:54.041410 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:42:54.042131 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:42:54.042134 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:42:54.042148 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:42:54.042221 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:54.042227 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:54.043225 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:54.043228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:54.044349 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:54.044358 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:54.045204 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:54.045208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:54.045368 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:54.045415 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 04:42:54.045595 (+ 180us) client_negotiation.cc:240] Negotiation successful
0925 04:42:54.045743 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":915,"spinlock_wait_cycles":7552,"thread_start_us":108,"threads_started":1}
I20260925 04:42:54.046129 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:54.040844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37549 (local address 127.0.0.1:52646)
0925 04:42:54.041208 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:54.041265 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 04:42:54.041410 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:42:54.042066 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:42:54.042069 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:42:54.042086 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:42:54.042175 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:54.042180 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:54.043720 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:54.043723 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:54.044676 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:54.044685 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:54.045769 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:54.045773 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:54.045852 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:54.045877 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 04:42:54.045919 (+ 42us) client_negotiation.cc:240] Negotiation successful
0925 04:42:54.045986 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":21,"spinlock_wait_cycles":9856,"thread_start_us":94,"threads_started":1}
I20260925 04:42:54.046751 18994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:54.041093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52646 (local address 127.0.98.1:37549)
0925 04:42:54.041690 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:54.041697 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:54.041716 (+ 19us) server_negotiation.cc:407] Connection header received
0925 04:42:54.041781 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:54.041786 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:54.041939 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:54.042365 (+ 426us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:42:54.042522 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:54.043585 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:54.044794 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:54.045639 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:54.046299 (+ 660us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:54.046462 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:54.046513 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 04:42:54.046610 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":57,"threads_started":1}
I20260925 04:42:54.047435 18990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:54.039989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33377 (local address 127.0.0.1:47450)
0925 04:42:54.041311 (+ 1322us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:54.041336 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 04:42:54.041464 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:42:54.041966 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:42:54.041972 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:42:54.041994 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:42:54.042130 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:54.042137 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:54.043586 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:54.043589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:54.045755 (+ 2166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:54.045763 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:54.047063 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:54.047066 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:54.047155 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:54.047176 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 04:42:54.047243 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 04:42:54.047311 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":916,"mutex_wait_us":23,"thread_start_us":96,"threads_started":1}
I20260925 04:42:54.047957 18991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:54.040128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47450 (local address 127.0.98.3:33377)
0925 04:42:54.040818 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:54.040831 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:54.041405 (+ 574us) server_negotiation.cc:407] Connection header received
0925 04:42:54.041587 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:54.041592 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:54.041797 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:54.041955 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:42:54.042248 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:54.043422 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:54.045891 (+ 2469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:54.046750 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:54.046862 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:54.047000 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:54.047686 (+ 686us) server_negotiation.cc:299] Negotiation successful
0925 04:42:54.047793 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":180,"threads_started":1}
I20260925 04:42:54.049633 18644 tablet_service.cc:1467] Processing CreateTablet for tablet 1cf4ca6ccea54214ac76f5f5e34be0c0 (DEFAULT_TABLE table=test-table [id=ac3d2f2774374e278d28c2dcf6607e5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:42:54.049743 18778 tablet_service.cc:1467] Processing CreateTablet for tablet 1cf4ca6ccea54214ac76f5f5e34be0c0 (DEFAULT_TABLE table=test-table [id=ac3d2f2774374e278d28c2dcf6607e5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:42:54.050870 18644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cf4ca6ccea54214ac76f5f5e34be0c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:42:54.051024 18778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cf4ca6ccea54214ac76f5f5e34be0c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:42:54.051098 18912 tablet_service.cc:1467] Processing CreateTablet for tablet 1cf4ca6ccea54214ac76f5f5e34be0c0 (DEFAULT_TABLE table=test-table [id=ac3d2f2774374e278d28c2dcf6607e5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:42:54.052278 18912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cf4ca6ccea54214ac76f5f5e34be0c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:42:54.058069 18995 tablet_bootstrap.cc:492] T 1cf4ca6ccea54214ac76f5f5e34be0c0 P 843bf034612d439caf70eb40bcc08b7c: Bootstrap starting.
I20260925 04:42:54.059743 18996 tablet_bootstrap.cc:492] T 1cf4ca6ccea54214ac76f5f5e34be0c0 P d4ac50adb8b346aa8600758dec15e523: Bootstrap starting.
I20260925 04:42:54.059765 18997 tablet_bootstrap.cc:492] T 1cf4ca6ccea54214ac76f5f5e34be0c0 P b92dc1f17c7b4d858081cddeb9612506: Bootstrap starting.
I20260925 04:42:54.061249 18995 tablet_bootstrap.cc:654] T 1cf4ca6ccea54214ac76f5f5e34be0c0 P 843bf034612d439caf70eb40bcc08b7c: Neither blocks nor log segments found. Creating new log.
I20260925 04:42:54.062088 18997 tablet_bootstrap.cc:654] T 1cf4ca6ccea54214ac76f5f5e34be0c0 P b92dc1f17c7b4d858081cddeb9612506: Neither blocks nor log segments found. Creating new log.
I20260925 04:42:54.062165 18995 log.cc:824] T 1cf4ca6ccea54214ac76f5f5e34be0c0 P 843bf034612d439caf70eb40bcc08b7c: Log is configured to *not* fsync() on all Append() calls
I20260925 04:42:54.062636 18996 tablet_bootstrap.cc:654] T 1cf4ca6ccea54214ac76f5f5e34be0c0 P d4ac50adb8b346aa8600758dec15e523: Neither blocks nor log segments found. Creating new log.
I20260925 04:42:54.063233 18997 log.cc:824] T 1cf4ca6ccea54214ac76f5f5e34be0c0 P b92dc1f17c7b4d858081cddeb9612506: Log is configured to *not* fsync() on all Append() calls
I20260925 04:42:54.063752 18996 log.cc:824] T 1cf4ca6ccea54214ac76f5f5e34be0c0 P d4ac50adb8b346aa8600758dec15e523: Log is configured to *not* fsync() on all Append() calls
I20260925 04:42:54.064502 18995 tablet_bootstrap.cc:492] T 1cf4ca6ccea54214ac76f5f5e34be0c0 P 843bf034612d439caf70eb40bcc08b7c: No bootstrap required, opened a new log
I20260925 04:42:54.064724 18995 ts_tablet_manager.cc:1397] T 1cf4ca6ccea54214ac76f5f5e34be0c0 P 843bf034612d439caf70eb40bcc08b7c: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 04:42:54.065222 18997 tablet_bootstrap.cc:492] T 1cf4ca6ccea54214ac76f5f5e34be0c0 P b92dc1f17c7b4d858081cddeb9612506: No bootstrap required, opened a new log
I20260925 04:42:54.065428 18997 ts_tablet_manager.cc:1397] T 1cf4ca6ccea54214ac76f5f5e34be0c0 P b92dc1f17c7b4d858081cddeb9612506: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 04:42:54.066115 18996 tablet_bootstrap.cc:492] T 1cf4ca6ccea54214ac76f5f5e34be0c0 P d4ac50adb8b346aa8600758dec15e523: No bootstrap required, opened a new log
I20260925 04:42:54.066301 18996 ts_tablet_manager.cc:1397] T 1cf4ca6ccea54214ac76f5f5e34be0c0 P d4ac50adb8b346aa8600758dec15e523: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260925 04:42:54.068969 18997 raft_consensus.cc:348] T 1cf4ca6ccea54214ac76f5f5e34be0c0 P b92dc1f17c7b4d858081cddeb9612506 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92dc1f17c7b4d858081cddeb9612506" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33377 } } peers { permanent_uuid: "d4ac50adb8b346aa8600758dec15e523" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38977 } } peers { permanent_uuid: "843bf034612d439caf70eb40bcc08b7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37549 } }
I20260925 04:42:54.069242 18997 raft_consensus.cc:374] T 1cf4ca6ccea54214ac76f5f5e34be0c0 P b92dc1f17c7b4d858081cddeb9612506 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:42:54.069293 18997 raft_consensus.cc:729] T 1cf4ca6ccea54214ac76f5f5e34be0c0 P b92dc1f17c7b4d858081cddeb9612506 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b92dc1f17c7b4d858081cddeb9612506, State: Initialized, Role: FOLLOWER
I20260925 04:42:54.069252 18995 raft_consensus.cc:348] T 1cf4ca6ccea54214ac76f5f5e34be0c0 P 843bf034612d439caf70eb40bcc08b7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92dc1f17c7b4d858081cddeb9612506" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33377 } } peers { permanent_uuid: "d4ac50adb8b346aa8600758dec15e523" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38977 } } peers { permanent_uuid: "843bf034612d439caf70eb40bcc08b7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37549 } }
I20260925 04:42:54.069595 18995 raft_consensus.cc:374] T 1cf4ca6ccea54214ac76f5f5e34be0c0 P 843bf034612d439caf70eb40bcc08b7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:42:54.069669 18995 raft_consensus.cc:729] T 1cf4ca6ccea54214ac76f5f5e34be0c0 P 843bf034612d439caf70eb40bcc08b7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 843bf034612d439caf70eb40bcc08b7c, State: Initialized, Role: FOLLOWER
I20260925 04:42:54.069814 18997 consensus_queue.cc:260] T 1cf4ca6ccea54214ac76f5f5e34be0c0 P b92dc1f17c7b4d858081cddeb9612506 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92dc1f17c7b4d858081cddeb9612506" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33377 } } peers { permanent_uuid: "d4ac50adb8b346aa8600758dec15e523" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38977 } } peers { permanent_uuid: "843bf034612d439caf70eb40bcc08b7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37549 } }
I20260925 04:42:54.070119 18995 consensus_queue.cc:260] T 1cf4ca6ccea54214ac76f5f5e34be0c0 P 843bf034612d439caf70eb40bcc08b7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92dc1f17c7b4d858081cddeb9612506" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33377 } } peers { permanent_uuid: "d4ac50adb8b346aa8600758dec15e523" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38977 } } peers { permanent_uuid: "843bf034612d439caf70eb40bcc08b7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37549 } }
I20260925 04:42:54.070080 18996 raft_consensus.cc:348] T 1cf4ca6ccea54214ac76f5f5e34be0c0 P d4ac50adb8b346aa8600758dec15e523 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92dc1f17c7b4d858081cddeb9612506" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33377 } } peers { permanent_uuid: "d4ac50adb8b346aa8600758dec15e523" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38977 } } peers { permanent_uuid: "843bf034612d439caf70eb40bcc08b7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37549 } }
I20260925 04:42:54.070364 18996 raft_consensus.cc:374] T 1cf4ca6ccea54214ac76f5f5e34be0c0 P d4ac50adb8b346aa8600758dec15e523 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:42:54.070484 18996 raft_consensus.cc:729] T 1cf4ca6ccea54214ac76f5f5e34be0c0 P d4ac50adb8b346aa8600758dec15e523 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4ac50adb8b346aa8600758dec15e523, State: Initialized, Role: FOLLOWER
I20260925 04:42:54.070732 18978 heartbeater.cc:503] Master 127.0.98.62:39081 was elected leader, sending a full tablet report...
I20260925 04:42:54.070958 18997 ts_tablet_manager.cc:1428] T 1cf4ca6ccea54214ac76f5f5e34be0c0 P b92dc1f17c7b4d858081cddeb9612506: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 04:42:54.070968 18996 consensus_queue.cc:260] T 1cf4ca6ccea54214ac76f5f5e34be0c0 P d4ac50adb8b346aa8600758dec15e523 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92dc1f17c7b4d858081cddeb9612506" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33377 } } peers { permanent_uuid: "d4ac50adb8b346aa8600758dec15e523" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38977 } } peers { permanent_uuid: "843bf034612d439caf70eb40bcc08b7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37549 } }
I20260925 04:42:54.071143 18995 ts_tablet_manager.cc:1428] T 1cf4ca6ccea54214ac76f5f5e34be0c0 P 843bf034612d439caf70eb40bcc08b7c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 04:42:54.071501 18710 heartbeater.cc:503] Master 127.0.98.62:39081 was elected leader, sending a full tablet report...
I20260925 04:42:54.071657 18844 heartbeater.cc:503] Master 127.0.98.62:39081 was elected leader, sending a full tablet report...
I20260925 04:42:54.073330 18996 ts_tablet_manager.cc:1428] T 1cf4ca6ccea54214ac76f5f5e34be0c0 P d4ac50adb8b346aa8600758dec15e523: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20260925 04:42:54.081143 19003 raft_consensus.cc:482] T 1cf4ca6ccea54214ac76f5f5e34be0c0 P d4ac50adb8b346aa8600758dec15e523 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:42:54.081333 19003 raft_consensus.cc:504] T 1cf4ca6ccea54214ac76f5f5e34be0c0 P d4ac50adb8b346aa8600758dec15e523 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92dc1f17c7b4d858081cddeb9612506" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33377 } } peers { permanent_uuid: "d4ac50adb8b346aa8600758dec15e523" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38977 } } peers { permanent_uuid: "843bf034612d439caf70eb40bcc08b7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37549 } }
I20260925 04:42:54.082419 19003 leader_election.cc:288] T 1cf4ca6ccea54214ac76f5f5e34be0c0 P d4ac50adb8b346aa8600758dec15e523 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b92dc1f17c7b4d858081cddeb9612506 (127.0.98.3:33377), 843bf034612d439caf70eb40bcc08b7c (127.0.98.1:37549)
I20260925 04:42:54.086187 18991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:54.082946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44171 (local address 127.0.98.3:33377)
0925 04:42:54.083065 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:54.083069 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:54.083083 (+ 14us) server_negotiation.cc:407] Connection header received
0925 04:42:54.083136 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:54.083139 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:54.083188 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:54.083280 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:42:54.083559 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:54.084111 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:54.085192 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:54.085622 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:54.085716 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:54.085783 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:54.085998 (+ 215us) server_negotiation.cc:299] Negotiation successful
0925 04:42:54.086071 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 04:42:54.086189 18994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:54.082886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41303 (local address 127.0.98.1:37549)
0925 04:42:54.083000 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:54.083004 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:54.083159 (+ 155us) server_negotiation.cc:407] Connection header received
0925 04:42:54.083276 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:54.083280 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:54.083331 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:54.083403 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:42:54.083666 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:54.084212 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:54.085225 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:54.085625 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:54.085717 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:54.085785 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:54.085981 (+ 196us) server_negotiation.cc:299] Negotiation successful
0925 04:42:54.086056 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 04:42:54.086189 18846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:54.082650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33377 (local address 127.0.98.2:44171)
0925 04:42:54.082779 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:54.082796 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:42:54.082893 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:42:54.083305 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:42:54.083308 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:42:54.083326 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:42:54.083455 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:54.083462 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:54.084214 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:54.084217 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:54.085076 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:54.085084 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:54.085734 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:54.085738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:54.085887 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:54.085904 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 04:42:54.085973 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 04:42:54.086064 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":58}
I20260925 04:42:54.086202 19004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:54.082650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37549 (local address 127.0.98.2:41303)
0925 04:42:54.083080 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:54.083092 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:42:54.083180 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:42:54.083424 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:42:54.083426 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:42:54.083440 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:42:54.083535 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:54.083541 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:54.084319 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:54.084321 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:54.085105 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:54.085113 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:54.085736 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:54.085740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:54.085873 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:54.085892 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 04:42:54.085948 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 04:42:54.086037 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"mutex_wait_us":56,"thread_start_us":133,"threads_started":1}
I20260925 04:42:54.086923 18932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cf4ca6ccea54214ac76f5f5e34be0c0" candidate_uuid: "d4ac50adb8b346aa8600758dec15e523" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b92dc1f17c7b4d858081cddeb9612506" is_pre_election: true
I20260925 04:42:54.087128 18664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cf4ca6ccea54214ac76f5f5e34be0c0" candidate_uuid: "d4ac50adb8b346aa8600758dec15e523" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "843bf034612d439caf70eb40bcc08b7c" is_pre_election: true
I20260925 04:42:54.087232 18932 raft_consensus.cc:2454] T 1cf4ca6ccea54214ac76f5f5e34be0c0 P b92dc1f17c7b4d858081cddeb9612506 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4ac50adb8b346aa8600758dec15e523 in term 0.
I20260925 04:42:54.087401 18664 raft_consensus.cc:2454] T 1cf4ca6ccea54214ac76f5f5e34be0c0 P 843bf034612d439caf70eb40bcc08b7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4ac50adb8b346aa8600758dec15e523 in term 0.
I20260925 04:42:54.087769 18732 leader_election.cc:302] T 1cf4ca6ccea54214ac76f5f5e34be0c0 P d4ac50adb8b346aa8600758dec15e523 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 843bf034612d439caf70eb40bcc08b7c, b92dc1f17c7b4d858081cddeb9612506, d4ac50adb8b346aa8600758dec15e523; no voters:
I20260925 04:42:54.088070 19003 raft_consensus.cc:2781] T 1cf4ca6ccea54214ac76f5f5e34be0c0 P d4ac50adb8b346aa8600758dec15e523 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:42:54.088191 19003 raft_consensus.cc:482] T 1cf4ca6ccea54214ac76f5f5e34be0c0 P d4ac50adb8b346aa8600758dec15e523 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:42:54.088249 19003 raft_consensus.cc:3037] T 1cf4ca6ccea54214ac76f5f5e34be0c0 P d4ac50adb8b346aa8600758dec15e523 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:42:54.089458 19003 raft_consensus.cc:504] T 1cf4ca6ccea54214ac76f5f5e34be0c0 P d4ac50adb8b346aa8600758dec15e523 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92dc1f17c7b4d858081cddeb9612506" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33377 } } peers { permanent_uuid: "d4ac50adb8b346aa8600758dec15e523" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38977 } } peers { permanent_uuid: "843bf034612d439caf70eb40bcc08b7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37549 } }
I20260925 04:42:54.089819 19003 leader_election.cc:288] T 1cf4ca6ccea54214ac76f5f5e34be0c0 P d4ac50adb8b346aa8600758dec15e523 [CANDIDATE]: Term 1 election: Requested vote from peers b92dc1f17c7b4d858081cddeb9612506 (127.0.98.3:33377), 843bf034612d439caf70eb40bcc08b7c (127.0.98.1:37549)
I20260925 04:42:54.090179 18932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cf4ca6ccea54214ac76f5f5e34be0c0" candidate_uuid: "d4ac50adb8b346aa8600758dec15e523" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b92dc1f17c7b4d858081cddeb9612506"
I20260925 04:42:54.090207 18664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cf4ca6ccea54214ac76f5f5e34be0c0" candidate_uuid: "d4ac50adb8b346aa8600758dec15e523" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "843bf034612d439caf70eb40bcc08b7c"
I20260925 04:42:54.090313 18932 raft_consensus.cc:3037] T 1cf4ca6ccea54214ac76f5f5e34be0c0 P b92dc1f17c7b4d858081cddeb9612506 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:42:54.090318 18664 raft_consensus.cc:3037] T 1cf4ca6ccea54214ac76f5f5e34be0c0 P 843bf034612d439caf70eb40bcc08b7c [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:42:54.091331 18664 raft_consensus.cc:2454] T 1cf4ca6ccea54214ac76f5f5e34be0c0 P 843bf034612d439caf70eb40bcc08b7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4ac50adb8b346aa8600758dec15e523 in term 1.
I20260925 04:42:54.091356 18932 raft_consensus.cc:2454] T 1cf4ca6ccea54214ac76f5f5e34be0c0 P b92dc1f17c7b4d858081cddeb9612506 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4ac50adb8b346aa8600758dec15e523 in term 1.
I20260925 04:42:54.091683 18734 leader_election.cc:302] T 1cf4ca6ccea54214ac76f5f5e34be0c0 P d4ac50adb8b346aa8600758dec15e523 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 843bf034612d439caf70eb40bcc08b7c, b92dc1f17c7b4d858081cddeb9612506, d4ac50adb8b346aa8600758dec15e523; no voters:
I20260925 04:42:54.091879 19003 raft_consensus.cc:2781] T 1cf4ca6ccea54214ac76f5f5e34be0c0 P d4ac50adb8b346aa8600758dec15e523 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:42:54.092092 19003 raft_consensus.cc:686] T 1cf4ca6ccea54214ac76f5f5e34be0c0 P d4ac50adb8b346aa8600758dec15e523 [term 1 LEADER]: Becoming Leader. State: Replica: d4ac50adb8b346aa8600758dec15e523, State: Running, Role: LEADER
I20260925 04:42:54.092413 19003 consensus_queue.cc:237] T 1cf4ca6ccea54214ac76f5f5e34be0c0 P d4ac50adb8b346aa8600758dec15e523 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92dc1f17c7b4d858081cddeb9612506" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33377 } } peers { permanent_uuid: "d4ac50adb8b346aa8600758dec15e523" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38977 } } peers { permanent_uuid: "843bf034612d439caf70eb40bcc08b7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37549 } }
I20260925 04:42:54.095587 18518 catalog_manager.cc:5495] T 1cf4ca6ccea54214ac76f5f5e34be0c0 P d4ac50adb8b346aa8600758dec15e523 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4ac50adb8b346aa8600758dec15e523 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "d4ac50adb8b346aa8600758dec15e523" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92dc1f17c7b4d858081cddeb9612506" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4ac50adb8b346aa8600758dec15e523" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "843bf034612d439caf70eb40bcc08b7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37549 } health_report { overall_health: UNKNOWN } } }
I20260925 04:42:54.107986 18992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:54.104421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55076 (local address 127.0.98.2:38977)
0925 04:42:54.104551 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:54.104556 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:54.104571 (+ 15us) server_negotiation.cc:407] Connection header received
0925 04:42:54.104715 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:54.104719 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:54.104778 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:54.104873 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:42:54.105100 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:54.105600 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:54.106422 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:54.106681 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:54.106774 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:54.107079 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:42:54.107512 (+ 433us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:42:54.107678 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:54.107792 (+ 114us) server_negotiation.cc:299] Negotiation successful
0925 04:42:54.107831 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 04:42:54.114254 18664 raft_consensus.cc:1261] T 1cf4ca6ccea54214ac76f5f5e34be0c0 P 843bf034612d439caf70eb40bcc08b7c [term 1 FOLLOWER]: Refusing update from remote peer d4ac50adb8b346aa8600758dec15e523: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:42:54.114249 18932 raft_consensus.cc:1261] T 1cf4ca6ccea54214ac76f5f5e34be0c0 P b92dc1f17c7b4d858081cddeb9612506 [term 1 FOLLOWER]: Refusing update from remote peer d4ac50adb8b346aa8600758dec15e523: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:42:54.114985 19003 consensus_queue.cc:1035] T 1cf4ca6ccea54214ac76f5f5e34be0c0 P d4ac50adb8b346aa8600758dec15e523 [LEADER]: Connected to new peer: Peer: permanent_uuid: "843bf034612d439caf70eb40bcc08b7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:42:54.115177 19005 consensus_queue.cc:1035] T 1cf4ca6ccea54214ac76f5f5e34be0c0 P d4ac50adb8b346aa8600758dec15e523 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92dc1f17c7b4d858081cddeb9612506" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:42:54.124959 19009 mvcc.cc:205] Tried to move back new op lower bound from 7333115388364599296 to 7333115388285894656. Current Snapshot: MvccSnapshot[applied={T|T < 7333115388364599296 or (T in {7333115388364599296})}]
I20260925 04:42:54.129040 19012 mvcc.cc:205] Tried to move back new op lower bound from 7333115388364599296 to 7333115388285894656. Current Snapshot: MvccSnapshot[applied={T|T < 7333115388364599296}]
I20260925 04:42:54.131932 18518 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:42:54.132138 18518 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:42:54.134840 18518 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 59e3272de5f440f7b62fe2418ea80e1b: Sending DeleteTablet for 3 replicas of tablet 1cf4ca6ccea54214ac76f5f5e34be0c0
I20260925 04:42:54.135732 18778 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cf4ca6ccea54214ac76f5f5e34be0c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:42:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55066
I20260925 04:42:54.135707 18912 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cf4ca6ccea54214ac76f5f5e34be0c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:42:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47450
I20260925 04:42:54.136255 19018 tablet_replica.cc:330] T 1cf4ca6ccea54214ac76f5f5e34be0c0 P d4ac50adb8b346aa8600758dec15e523: stopping tablet replica
I20260925 04:42:54.136651 19018 raft_consensus.cc:2229] T 1cf4ca6ccea54214ac76f5f5e34be0c0 P d4ac50adb8b346aa8600758dec15e523 [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:42:54.136636 18644 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cf4ca6ccea54214ac76f5f5e34be0c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:42:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52646
I20260925 04:42:54.137006 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 18581
I20260925 04:42:54.137171 19018 raft_consensus.cc:2258] T 1cf4ca6ccea54214ac76f5f5e34be0c0 P d4ac50adb8b346aa8600758dec15e523 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:42:54.139989 19018 ts_tablet_manager.cc:1905] T 1cf4ca6ccea54214ac76f5f5e34be0c0 P d4ac50adb8b346aa8600758dec15e523: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:42:54.143154 19018 ts_tablet_manager.cc:1918] T 1cf4ca6ccea54214ac76f5f5e34be0c0 P d4ac50adb8b346aa8600758dec15e523: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:42:54.143289 19018 log.cc:1196] T 1cf4ca6ccea54214ac76f5f5e34be0c0 P d4ac50adb8b346aa8600758dec15e523: Deleting WAL directory at /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/minicluster-data/ts-1/wal/wals/1cf4ca6ccea54214ac76f5f5e34be0c0
I20260925 04:42:54.143676 19019 tablet_replica.cc:330] T 1cf4ca6ccea54214ac76f5f5e34be0c0 P b92dc1f17c7b4d858081cddeb9612506: stopping tablet replica
I20260925 04:42:54.143643 19018 ts_tablet_manager.cc:1939] T 1cf4ca6ccea54214ac76f5f5e34be0c0 P d4ac50adb8b346aa8600758dec15e523: Deleting consensus metadata
I20260925 04:42:54.144146 19019 raft_consensus.cc:2229] T 1cf4ca6ccea54214ac76f5f5e34be0c0 P b92dc1f17c7b4d858081cddeb9612506 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:42:54.144537 19019 raft_consensus.cc:2258] T 1cf4ca6ccea54214ac76f5f5e34be0c0 P b92dc1f17c7b4d858081cddeb9612506 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:42:54.145071 18506 catalog_manager.cc:4841] TS d4ac50adb8b346aa8600758dec15e523 (127.0.98.2:38977): tablet 1cf4ca6ccea54214ac76f5f5e34be0c0 (table test-table [id=ac3d2f2774374e278d28c2dcf6607e5e]) successfully deleted
W20260925 04:42:54.145728 18506 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37549 (error 108)
I20260925 04:42:54.145794 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 18715
I20260925 04:42:54.148072 19019 ts_tablet_manager.cc:1905] T 1cf4ca6ccea54214ac76f5f5e34be0c0 P b92dc1f17c7b4d858081cddeb9612506: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:42:54.148563 18990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:54.148206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37549 (local address 127.0.0.1:52650)
0925 04:42:54.148393 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:54.148454 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
W20260925 04:42:54.148802 18506 catalog_manager.cc:4584] TS 843bf034612d439caf70eb40bcc08b7c (127.0.98.1:37549): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1cf4ca6ccea54214ac76f5f5e34be0c0: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37549: connect: Connection refused (error 111)
I20260925 04:42:54.151196 19019 ts_tablet_manager.cc:1918] T 1cf4ca6ccea54214ac76f5f5e34be0c0 P b92dc1f17c7b4d858081cddeb9612506: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:42:54.151372 19019 log.cc:1196] T 1cf4ca6ccea54214ac76f5f5e34be0c0 P b92dc1f17c7b4d858081cddeb9612506: Deleting WAL directory at /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311123478858-392-0/minicluster-data/ts-2/wal/wals/1cf4ca6ccea54214ac76f5f5e34be0c0
I20260925 04:42:54.151885 19019 ts_tablet_manager.cc:1939] T 1cf4ca6ccea54214ac76f5f5e34be0c0 P b92dc1f17c7b4d858081cddeb9612506: Deleting consensus metadata
I20260925 04:42:54.152873 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 18849
I20260925 04:42:54.157841 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 18487
2026-09-25T04:42:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2239 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-25T04:42:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:42:54Z Disabled control of system clock
I20260925 04:42:54.187142 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311123478858-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311123478858-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311123478858-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42037
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42037
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 04:42:54.299995 19025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:42:54.300292 19025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:42:54.300328 19025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:42:54.304917 19025 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:42:54.305013 19025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:42:54.305027 19025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:42:54.305044 19025 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:42:54.305059 19025 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:42:54.310357 19025 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311123478858-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311123478858-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42037
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311123478858-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:42:54.311497 19025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:42:54.313048 19025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:42:54.321841 19030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:42:54.321856 19031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:42:54.321928 19025 server_base.cc:1034] running on GCE node
W20260925 04:42:54.321851 19033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:42:54.322790 19025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:42:54.324007 19025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:42:54.325196 19025 hybrid_clock.cc:648] HybridClock initialized: now 1790311374325170 us; error 43 us; skew 500 ppm
I20260925 04:42:54.327152 19025 webserver.cc:460] Webserver started at http://127.0.98.62:39257/ using document root <none> and password file <none>
I20260925 04:42:54.327850 19025 fs_manager.cc:359] Metadata directory not provided
I20260925 04:42:54.327925 19025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:42:54.328141 19025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:42:54.330297 19025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311123478858-392-0/minicluster-data/master-0/data/instance:
uuid: "e2cb7fe70f244423b604b679c3d5bbbe"
format_stamp: "Formatted at 2026-09-25 04:42:54 on dist-test-slave-dc95"
server_key: "87715eef511d12d0b09c35e25abbfcab"
server_key_iv: "58b36873df61bb478e057f8420cbaf76"
server_key_version: "encryptionkey@0"
I20260925 04:42:54.330893 19025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311123478858-392-0/minicluster-data/master-0/wal/instance:
uuid: "e2cb7fe70f244423b604b679c3d5bbbe"
format_stamp: "Formatted at 2026-09-25 04:42:54 on dist-test-slave-dc95"
server_key: "87715eef511d12d0b09c35e25abbfcab"
server_key_iv: "58b36873df61bb478e057f8420cbaf76"
server_key_version: "encryptionkey@0"
I20260925 04:42:54.334849 19025 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 04:42:54.337543 19039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:54.338912 19025 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:42:54.339035 19025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311123478858-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311123478858-392-0/minicluster-data/master-0/wal
uuid: "e2cb7fe70f244423b604b679c3d5bbbe"
format_stamp: "Formatted at 2026-09-25 04:42:54 on dist-test-slave-dc95"
server_key: "87715eef511d12d0b09c35e25abbfcab"
server_key_iv: "58b36873df61bb478e057f8420cbaf76"
server_key_version: "encryptionkey@0"
I20260925 04:42:54.339144 19025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311123478858-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311123478858-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311123478858-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:42:54.365907 19025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:42:54.366638 19025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:42:54.366835 19025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:42:54.373895 19025 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42037
I20260925 04:42:54.373912 19091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42037 every 8 connection(s)
I20260925 04:42:54.374895 19025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
I20260925 04:42:54.378229 19092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:42:54.383272 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 19025
I20260925 04:42:54.383414 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311123478858-392-0/minicluster-data/master-0/wal/instance
I20260925 04:42:54.383715 392 external_mini_cluster.cc:1428] Setting key ad5b74c57b3738fa9ab61fc87091d681
I20260925 04:42:54.384253 19092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2cb7fe70f244423b604b679c3d5bbbe: Bootstrap starting.
I20260925 04:42:54.386749 19092 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2cb7fe70f244423b604b679c3d5bbbe: Neither blocks nor log segments found. Creating new log.
I20260925 04:42:54.387647 19092 log.cc:824] T 00000000000000000000000000000000 P e2cb7fe70f244423b604b679c3d5bbbe: Log is configured to *not* fsync() on all Append() calls
I20260925 04:42:54.389904 19092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2cb7fe70f244423b604b679c3d5bbbe: No bootstrap required, opened a new log
I20260925 04:42:54.393265 19095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:54.385079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38904 (local address 127.0.98.62:42037)
0925 04:42:54.385744 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:54.385758 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:54.385796 (+ 38us) server_negotiation.cc:407] Connection header received
0925 04:42:54.386497 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:54.386516 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:54.386968 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:54.387281 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:42:54.387689 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:54.388827 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:54.389995 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:54.390485 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:54.390623 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:54.391460 (+ 837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:42:54.391466 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:42:54.391478 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:42:54.391939 (+ 461us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:42:54.391997 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:42:54.392018 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:42:54.392096 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:54.392243 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 04:42:54.392527 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":108,"threads_started":1}
I20260925 04:42:54.393854 19092 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2cb7fe70f244423b604b679c3d5bbbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2cb7fe70f244423b604b679c3d5bbbe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42037 } }
I20260925 04:42:54.394078 19092 raft_consensus.cc:374] T 00000000000000000000000000000000 P e2cb7fe70f244423b604b679c3d5bbbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:42:54.394115 19092 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2cb7fe70f244423b604b679c3d5bbbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2cb7fe70f244423b604b679c3d5bbbe, State: Initialized, Role: FOLLOWER
I20260925 04:42:54.394536 19092 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2cb7fe70f244423b604b679c3d5bbbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2cb7fe70f244423b604b679c3d5bbbe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42037 } }
I20260925 04:42:54.394663 19092 raft_consensus.cc:388] T 00000000000000000000000000000000 P e2cb7fe70f244423b604b679c3d5bbbe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:42:54.394721 19092 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2cb7fe70f244423b604b679c3d5bbbe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:42:54.394812 19092 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2cb7fe70f244423b604b679c3d5bbbe [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:42:54.395735 19092 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2cb7fe70f244423b604b679c3d5bbbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2cb7fe70f244423b604b679c3d5bbbe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42037 } }
I20260925 04:42:54.396050 19092 leader_election.cc:302] T 00000000000000000000000000000000 P e2cb7fe70f244423b604b679c3d5bbbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e2cb7fe70f244423b604b679c3d5bbbe; no voters:
I20260925 04:42:54.396330 19092 leader_election.cc:288] T 00000000000000000000000000000000 P e2cb7fe70f244423b604b679c3d5bbbe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:42:54.396438 19097 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2cb7fe70f244423b604b679c3d5bbbe [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:42:54.396646 19097 raft_consensus.cc:686] T 00000000000000000000000000000000 P e2cb7fe70f244423b604b679c3d5bbbe [term 1 LEADER]: Becoming Leader. State: Replica: e2cb7fe70f244423b604b679c3d5bbbe, State: Running, Role: LEADER
I20260925 04:42:54.396957 19097 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2cb7fe70f244423b604b679c3d5bbbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2cb7fe70f244423b604b679c3d5bbbe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42037 } }
I20260925 04:42:54.397444 19092 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2cb7fe70f244423b604b679c3d5bbbe [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:42:54.398444 19098 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2cb7fe70f244423b604b679c3d5bbbe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e2cb7fe70f244423b604b679c3d5bbbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2cb7fe70f244423b604b679c3d5bbbe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42037 } } }
I20260925 04:42:54.398561 19098 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2cb7fe70f244423b604b679c3d5bbbe [sys.catalog]: This master's current role is: LEADER
I20260925 04:42:54.398826 19099 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2cb7fe70f244423b604b679c3d5bbbe [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2cb7fe70f244423b604b679c3d5bbbe. Latest consensus state: current_term: 1 leader_uuid: "e2cb7fe70f244423b604b679c3d5bbbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2cb7fe70f244423b604b679c3d5bbbe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42037 } } }
I20260925 04:42:54.398905 19099 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2cb7fe70f244423b604b679c3d5bbbe [sys.catalog]: This master's current role is: LEADER
I20260925 04:42:54.399561 19106 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:42:54.402371 19106 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:42:54.407516 19106 catalog_manager.cc:1347] Generated new cluster ID: 4018831e78b9428b8d04d8cf2f27f730
I20260925 04:42:54.407680 19106 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:42:54.427492 19106 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:42:54.428931 19106 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:42:54.441182 19106 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e2cb7fe70f244423b604b679c3d5bbbe: Generated new TSK 0
I20260925 04:42:54.442276 19106 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:42:54.446265 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311123478858-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311123478858-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311123478858-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42037
--builtin_ntp_servers=127.0.98.20:43939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 04:42:54.562003 19116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:42:54.562294 19116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:42:54.562333 19116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:42:54.566543 19116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:42:54.566622 19116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:42:54.566679 19116 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 04:42:54.571744 19116 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311123478858-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311123478858-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311123478858-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:42:54.572937 19116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:42:54.574286 19116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:42:54.584213 19122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:42:54.584357 19116 server_base.cc:1034] running on GCE node
W20260925 04:42:54.584256 19121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:42:54.584599 19124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:42:54.585018 19116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:42:54.585779 19116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:42:54.586969 19116 hybrid_clock.cc:648] HybridClock initialized: now 1790311374586951 us; error 37 us; skew 500 ppm
I20260925 04:42:54.589102 19116 webserver.cc:460] Webserver started at http://127.0.98.1:43639/ using document root <none> and password file <none>
I20260925 04:42:54.589712 19116 fs_manager.cc:359] Metadata directory not provided
I20260925 04:42:54.589782 19116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:42:54.589990 19116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:42:54.592262 19116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311123478858-392-0/minicluster-data/ts-0/data/instance:
uuid: "d2299066cb27477e9bd5ef207ff9a1dc"
format_stamp: "Formatted at 2026-09-25 04:42:54 on dist-test-slave-dc95"
server_key: "e1884c48699888309b653d683992e9ae"
server_key_iv: "1d8b7880574a50b30d5d564084c235a9"
server_key_version: "encryptionkey@0"
I20260925 04:42:54.592859 19116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311123478858-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d2299066cb27477e9bd5ef207ff9a1dc"
format_stamp: "Formatted at 2026-09-25 04:42:54 on dist-test-slave-dc95"
server_key: "e1884c48699888309b653d683992e9ae"
server_key_iv: "1d8b7880574a50b30d5d564084c235a9"
server_key_version: "encryptionkey@0"
I20260925 04:42:54.596724 19116 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 04:42:54.599565 19130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:54.601079 19116 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 04:42:54.601210 19116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311123478858-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311123478858-392-0/minicluster-data/ts-0/wal
uuid: "d2299066cb27477e9bd5ef207ff9a1dc"
format_stamp: "Formatted at 2026-09-25 04:42:54 on dist-test-slave-dc95"
server_key: "e1884c48699888309b653d683992e9ae"
server_key_iv: "1d8b7880574a50b30d5d564084c235a9"
server_key_version: "encryptionkey@0"
I20260925 04:42:54.601346 19116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311123478858-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311123478858-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311123478858-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:42:54.615258 19116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:42:54.616074 19116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:42:54.616276 19116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:42:54.616901 19116 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:42:54.617934 19116 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:42:54.617990 19116 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:54.618045 19116 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:42:54.618069 19116 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:54.626772 19116 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41571
I20260925 04:42:54.626809 19243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41571 every 8 connection(s)
I20260925 04:42:54.627767 19116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
I20260925 04:42:54.632712 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 19116
I20260925 04:42:54.632846 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311123478858-392-0/minicluster-data/ts-0/wal/instance
I20260925 04:42:54.633143 392 external_mini_cluster.cc:1428] Setting key cba2666243b2a21ab14f174213b8c384
I20260925 04:42:54.635212 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311123478858-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311123478858-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311123478858-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42037
--builtin_ntp_servers=127.0.98.20:43939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 04:42:54.637903 19095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:54.630002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57385 (local address 127.0.98.62:42037)
0925 04:42:54.630186 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:54.630191 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:54.631440 (+ 1249us) server_negotiation.cc:407] Connection header received
0925 04:42:54.632922 (+ 1482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:54.632925 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:54.632986 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:54.633056 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:42:54.634136 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:54.634766 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:54.636011 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:54.636251 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:54.636342 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:54.637449 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:42:54.637452 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:42:54.637454 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:42:54.637466 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:42:54.637486 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:42:54.637494 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:42:54.637558 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:54.637695 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 04:42:54.637760 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 04:42:54.638866 19246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:54.630542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42037 (local address 127.0.98.1:57385)
0925 04:42:54.631279 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:54.631323 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 04:42:54.632682 (+ 1359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:42:54.633277 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:42:54.633288 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:42:54.633383 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:42:54.633983 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:54.633998 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:54.634887 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:54.634890 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:54.635890 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:54.635903 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:54.636401 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:54.636405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:54.636573 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:54.637172 (+ 599us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:42:54.637173 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:42:54.637219 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:42:54.637295 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:42:54.637563 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:42:54.637573 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:42:54.637579 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 04:42:54.637691 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 04:42:54.638099 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":142,"threads_started":1}
I20260925 04:42:54.640237 19244 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42037
I20260925 04:42:54.640522 19244 heartbeater.cc:464] Registering TS with master...
I20260925 04:42:54.641211 19244 heartbeater.cc:511] Master 127.0.98.62:42037 requested a full tablet report, sending...
I20260925 04:42:54.643172 19056 ts_manager.cc:194] Registered new tserver with Master: d2299066cb27477e9bd5ef207ff9a1dc (127.0.98.1:41571)
I20260925 04:42:54.644464 19056 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:57385
W20260925 04:42:54.748330 19247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:42:54.748615 19247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:42:54.748656 19247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:42:54.752836 19247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:42:54.752908 19247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:42:54.752967 19247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 04:42:54.757905 19247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311123478858-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311123478858-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311123478858-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:42:54.759055 19247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:42:54.760530 19247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:42:54.770095 19252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:42:54.770107 19253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:42:54.770164 19247 server_base.cc:1034] running on GCE node
W20260925 04:42:54.770110 19255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:42:54.770694 19247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:42:54.771394 19247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:42:54.772558 19247 hybrid_clock.cc:648] HybridClock initialized: now 1790311374772542 us; error 40 us; skew 500 ppm
I20260925 04:42:54.774375 19247 webserver.cc:460] Webserver started at http://127.0.98.2:44543/ using document root <none> and password file <none>
I20260925 04:42:54.774925 19247 fs_manager.cc:359] Metadata directory not provided
I20260925 04:42:54.775003 19247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:42:54.775203 19247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:42:54.777611 19247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311123478858-392-0/minicluster-data/ts-1/data/instance:
uuid: "a1b0fa80719745d7b975b3ba1d5449de"
format_stamp: "Formatted at 2026-09-25 04:42:54 on dist-test-slave-dc95"
server_key: "8b5fff00a2a94e9075e79649814bf31d"
server_key_iv: "bebfc9f8f5e19f8e790e8361322566b8"
server_key_version: "encryptionkey@0"
I20260925 04:42:54.778225 19247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311123478858-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a1b0fa80719745d7b975b3ba1d5449de"
format_stamp: "Formatted at 2026-09-25 04:42:54 on dist-test-slave-dc95"
server_key: "8b5fff00a2a94e9075e79649814bf31d"
server_key_iv: "bebfc9f8f5e19f8e790e8361322566b8"
server_key_version: "encryptionkey@0"
I20260925 04:42:54.782447 19247 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260925 04:42:54.785076 19261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:54.786422 19247 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 04:42:54.786556 19247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311123478858-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311123478858-392-0/minicluster-data/ts-1/wal
uuid: "a1b0fa80719745d7b975b3ba1d5449de"
format_stamp: "Formatted at 2026-09-25 04:42:54 on dist-test-slave-dc95"
server_key: "8b5fff00a2a94e9075e79649814bf31d"
server_key_iv: "bebfc9f8f5e19f8e790e8361322566b8"
server_key_version: "encryptionkey@0"
I20260925 04:42:54.786674 19247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311123478858-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311123478858-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311123478858-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:42:54.828609 19247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:42:54.829345 19247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:42:54.829528 19247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:42:54.830138 19247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:42:54.831113 19247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:42:54.831162 19247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:54.831215 19247 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:42:54.831243 19247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:54.839664 19247 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46295
I20260925 04:42:54.839707 19374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46295 every 8 connection(s)
I20260925 04:42:54.840575 19247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
I20260925 04:42:54.841991 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 19247
I20260925 04:42:54.842100 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311123478858-392-0/minicluster-data/ts-1/wal/instance
I20260925 04:42:54.842383 392 external_mini_cluster.cc:1428] Setting key a175d52a888364ba5fcdbc63ab61d937
I20260925 04:42:54.844506 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311123478858-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311123478858-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311123478858-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42037
--builtin_ntp_servers=127.0.98.20:43939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 04:42:54.850814 19095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:54.842553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59379 (local address 127.0.98.62:42037)
0925 04:42:54.842686 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:54.842691 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:54.843908 (+ 1217us) server_negotiation.cc:407] Connection header received
0925 04:42:54.845370 (+ 1462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:54.845373 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:54.845427 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:54.845507 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:42:54.846534 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:54.847292 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:54.848627 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:54.848974 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:54.849093 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:54.850188 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:42:54.850192 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:42:54.850195 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:42:54.850213 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:42:54.850238 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:42:54.850249 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:42:54.850329 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:54.850575 (+ 246us) server_negotiation.cc:299] Negotiation successful
0925 04:42:54.850650 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 04:42:54.851742 19377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:54.842977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42037 (local address 127.0.98.2:59379)
0925 04:42:54.843769 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:54.843825 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 04:42:54.845166 (+ 1341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:42:54.845773 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:42:54.845780 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:42:54.845887 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:42:54.846395 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:54.846409 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:54.847436 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:54.847438 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:54.848499 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:54.848511 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:54.849118 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:54.849120 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:54.849278 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:54.849907 (+ 629us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:42:54.849908 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:42:54.849954 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:42:54.850029 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:42:54.850377 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:42:54.850386 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:42:54.850391 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 04:42:54.850574 (+ 183us) client_negotiation.cc:240] Negotiation successful
0925 04:42:54.850832 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":121,"threads_started":1}
I20260925 04:42:54.852948 19375 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42037
I20260925 04:42:54.853221 19375 heartbeater.cc:464] Registering TS with master...
I20260925 04:42:54.853915 19375 heartbeater.cc:511] Master 127.0.98.62:42037 requested a full tablet report, sending...
I20260925 04:42:54.855067 19056 ts_manager.cc:194] Registered new tserver with Master: a1b0fa80719745d7b975b3ba1d5449de (127.0.98.2:46295)
I20260925 04:42:54.855734 19056 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:59379
W20260925 04:42:54.958304 19378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:42:54.958612 19378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:42:54.958653 19378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:42:54.962975 19378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:42:54.963053 19378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:42:54.963114 19378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 04:42:54.968138 19378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311123478858-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311123478858-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19378
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311123478858-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:42:54.969274 19378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:42:54.970741 19378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:42:54.980334 19384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:42:54.980414 19386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:42:54.980334 19383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:42:54.980530 19378 server_base.cc:1034] running on GCE node
I20260925 04:42:54.981267 19378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:42:54.981940 19378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:42:54.983114 19378 hybrid_clock.cc:648] HybridClock initialized: now 1790311374983092 us; error 33 us; skew 500 ppm
I20260925 04:42:54.985038 19378 webserver.cc:460] Webserver started at http://127.0.98.3:34073/ using document root <none> and password file <none>
I20260925 04:42:54.985610 19378 fs_manager.cc:359] Metadata directory not provided
I20260925 04:42:54.985678 19378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:42:54.985877 19378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:42:54.988075 19378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311123478858-392-0/minicluster-data/ts-2/data/instance:
uuid: "dffa177b4cb94f03b151246e5b28398e"
format_stamp: "Formatted at 2026-09-25 04:42:54 on dist-test-slave-dc95"
server_key: "258ea838d0c3b4a5ca3a113b584788b1"
server_key_iv: "7611a646f4ecdae7442ab09f149d57ea"
server_key_version: "encryptionkey@0"
I20260925 04:42:54.988636 19378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311123478858-392-0/minicluster-data/ts-2/wal/instance:
uuid: "dffa177b4cb94f03b151246e5b28398e"
format_stamp: "Formatted at 2026-09-25 04:42:54 on dist-test-slave-dc95"
server_key: "258ea838d0c3b4a5ca3a113b584788b1"
server_key_iv: "7611a646f4ecdae7442ab09f149d57ea"
server_key_version: "encryptionkey@0"
I20260925 04:42:54.992624 19378 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 04:42:54.995260 19392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:54.996584 19378 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 04:42:54.996716 19378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311123478858-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311123478858-392-0/minicluster-data/ts-2/wal
uuid: "dffa177b4cb94f03b151246e5b28398e"
format_stamp: "Formatted at 2026-09-25 04:42:54 on dist-test-slave-dc95"
server_key: "258ea838d0c3b4a5ca3a113b584788b1"
server_key_iv: "7611a646f4ecdae7442ab09f149d57ea"
server_key_version: "encryptionkey@0"
I20260925 04:42:54.996855 19378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311123478858-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311123478858-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311123478858-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:42:55.016934 19378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:42:55.017676 19378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:42:55.017876 19378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:42:55.018528 19378 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:42:55.019531 19378 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:42:55.019580 19378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:55.019680 19378 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:42:55.019706 19378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:55.028389 19378 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34587
I20260925 04:42:55.028421 19505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34587 every 8 connection(s)
I20260925 04:42:55.029353 19378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
I20260925 04:42:55.030967 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 19378
I20260925 04:42:55.031060 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311123478858-392-0/minicluster-data/ts-2/wal/instance
I20260925 04:42:55.031356 392 external_mini_cluster.cc:1428] Setting key 0fa48212fae99e8fe0103b11726da29b
I20260925 04:42:55.039562 19095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:55.031295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44775 (local address 127.0.98.62:42037)
0925 04:42:55.031455 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:55.031460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:55.032716 (+ 1256us) server_negotiation.cc:407] Connection header received
0925 04:42:55.034148 (+ 1432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:55.034151 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:55.034210 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:55.034332 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:42:55.035416 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:55.036019 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:55.037536 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:55.037893 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:55.037984 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:55.039105 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:42:55.039109 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:42:55.039111 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:42:55.039125 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:42:55.039146 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:42:55.039155 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:42:55.039227 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:55.039378 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 04:42:55.039438 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 04:42:55.040462 19508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:55.031805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42037 (local address 127.0.98.3:44775)
0925 04:42:55.032576 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:55.032632 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 04:42:55.033927 (+ 1295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:42:55.034523 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:42:55.034532 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:42:55.034663 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:42:55.035216 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:55.035230 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:55.036170 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:55.036173 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:55.037393 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:55.037406 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:55.038040 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:55.038044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:55.038199 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:55.038816 (+ 617us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:42:55.038818 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:42:55.038862 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:42:55.038944 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:42:55.039228 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:42:55.039232 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:42:55.039236 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 04:42:55.039374 (+ 138us) client_negotiation.cc:240] Negotiation successful
0925 04:42:55.039651 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":121,"threads_started":1}
I20260925 04:42:55.041669 19506 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42037
I20260925 04:42:55.041914 19506 heartbeater.cc:464] Registering TS with master...
I20260925 04:42:55.042500 19506 heartbeater.cc:511] Master 127.0.98.62:42037 requested a full tablet report, sending...
I20260925 04:42:55.043673 19056 ts_manager.cc:194] Registered new tserver with Master: dffa177b4cb94f03b151246e5b28398e (127.0.98.3:34587)
I20260925 04:42:55.044385 19056 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:44775
I20260925 04:42:55.045115 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:42:55.050788 19095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:55.046967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38914 (local address 127.0.98.62:42037)
0925 04:42:55.047101 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:55.047105 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:55.047226 (+ 121us) server_negotiation.cc:407] Connection header received
0925 04:42:55.047337 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:55.047340 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:55.047421 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:55.047485 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:42:55.047783 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:55.048294 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:55.049109 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:55.049373 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:55.049462 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:55.050304 (+ 842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:42:55.050307 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:42:55.050309 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:42:55.050321 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:42:55.050339 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:42:55.050355 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:42:55.050457 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:55.050593 (+ 136us) server_negotiation.cc:299] Negotiation successful
0925 04:42:55.050637 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 04:42:55.054455 19055 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38914:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:42:55.057687 19055 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:42:55.072515 19517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:55.066695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41571 (local address 127.0.0.1:42556)
0925 04:42:55.067054 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:55.067106 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 04:42:55.067239 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:42:55.068164 (+ 925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:42:55.068169 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:42:55.068191 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:42:55.068329 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:55.068337 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:55.069603 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:55.069606 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:55.070925 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:55.070934 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:55.071974 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:55.071977 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:55.072081 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:55.072118 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 04:42:55.072239 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 04:42:55.072360 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":86,"threads_started":1}
I20260925 04:42:55.072521 19518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:55.066849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42556 (local address 127.0.98.1:41571)
0925 04:42:55.067204 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:55.067215 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:55.067477 (+ 262us) server_negotiation.cc:407] Connection header received
0925 04:42:55.067565 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:55.067572 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:55.067834 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:55.067989 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:42:55.068451 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:55.069507 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:55.071060 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:55.071858 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:55.071952 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:55.072148 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:55.072267 (+ 119us) server_negotiation.cc:299] Negotiation successful
0925 04:42:55.072366 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":136,"threads_started":1}
I20260925 04:42:55.072901 19519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:55.067697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46295 (local address 127.0.0.1:59308)
0925 04:42:55.068210 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:55.068222 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:42:55.068288 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:42:55.069139 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:42:55.069142 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:42:55.069302 (+ 160us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:42:55.069404 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:55.069412 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:55.070351 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:55.070355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:55.071357 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:55.071366 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:55.072532 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:55.072535 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:55.072614 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:55.072635 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 04:42:55.072697 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 04:42:55.072763 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"mutex_wait_us":27,"thread_start_us":86,"threads_started":1}
I20260925 04:42:55.073155 19521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:55.066694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34587 (local address 127.0.0.1:35018)
0925 04:42:55.067984 (+ 1290us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:55.068000 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:42:55.068167 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:42:55.068756 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:42:55.068758 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:42:55.068771 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:42:55.068936 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:55.068946 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:55.070397 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:55.070400 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:55.071549 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:55.071558 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:55.072767 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:55.072770 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:55.072871 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:55.072892 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 04:42:55.072967 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 04:42:55.073033 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1156,"mutex_wait_us":654,"thread_start_us":76,"threads_started":1}
I20260925 04:42:55.073292 19520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:55.067260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35018 (local address 127.0.98.3:34587)
0925 04:42:55.067775 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:55.067785 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:55.068096 (+ 311us) server_negotiation.cc:407] Connection header received
0925 04:42:55.068311 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:55.068316 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:55.068497 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:55.068618 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:42:55.069142 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:55.070260 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:55.071684 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:55.072450 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:55.072547 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:55.072715 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:55.073031 (+ 316us) server_negotiation.cc:299] Negotiation successful
0925 04:42:55.073132 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":181,"threads_started":1}
I20260925 04:42:55.073375 19522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:55.067833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59308 (local address 127.0.98.2:46295)
0925 04:42:55.068669 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:55.068679 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:55.068700 (+ 21us) server_negotiation.cc:407] Connection header received
0925 04:42:55.068769 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:55.068774 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:55.068943 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:55.069075 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:42:55.069529 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:55.070239 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:55.071455 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:55.072170 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:55.072285 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:55.072469 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:55.073107 (+ 638us) server_negotiation.cc:299] Negotiation successful
0925 04:42:55.073206 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":736,"thread_start_us":115,"threads_started":1}
I20260925 04:42:55.075886 19440 tablet_service.cc:1467] Processing CreateTablet for tablet a081b7f33c36406da49440781a46e7dd (DEFAULT_TABLE table=test-table [id=d198caf5e51c44d88dcd147de6592094]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:42:55.075898 19309 tablet_service.cc:1467] Processing CreateTablet for tablet a081b7f33c36406da49440781a46e7dd (DEFAULT_TABLE table=test-table [id=d198caf5e51c44d88dcd147de6592094]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:42:55.075898 19178 tablet_service.cc:1467] Processing CreateTablet for tablet a081b7f33c36406da49440781a46e7dd (DEFAULT_TABLE table=test-table [id=d198caf5e51c44d88dcd147de6592094]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:42:55.076833 19440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a081b7f33c36406da49440781a46e7dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:42:55.076906 19178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a081b7f33c36406da49440781a46e7dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:42:55.076906 19309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a081b7f33c36406da49440781a46e7dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:42:55.083546 19523 tablet_bootstrap.cc:492] T a081b7f33c36406da49440781a46e7dd P dffa177b4cb94f03b151246e5b28398e: Bootstrap starting.
I20260925 04:42:55.084530 19524 tablet_bootstrap.cc:492] T a081b7f33c36406da49440781a46e7dd P a1b0fa80719745d7b975b3ba1d5449de: Bootstrap starting.
I20260925 04:42:55.084839 19525 tablet_bootstrap.cc:492] T a081b7f33c36406da49440781a46e7dd P d2299066cb27477e9bd5ef207ff9a1dc: Bootstrap starting.
I20260925 04:42:55.086073 19523 tablet_bootstrap.cc:654] T a081b7f33c36406da49440781a46e7dd P dffa177b4cb94f03b151246e5b28398e: Neither blocks nor log segments found. Creating new log.
I20260925 04:42:55.086531 19525 tablet_bootstrap.cc:654] T a081b7f33c36406da49440781a46e7dd P d2299066cb27477e9bd5ef207ff9a1dc: Neither blocks nor log segments found. Creating new log.
I20260925 04:42:55.086581 19524 tablet_bootstrap.cc:654] T a081b7f33c36406da49440781a46e7dd P a1b0fa80719745d7b975b3ba1d5449de: Neither blocks nor log segments found. Creating new log.
I20260925 04:42:55.086966 19523 log.cc:824] T a081b7f33c36406da49440781a46e7dd P dffa177b4cb94f03b151246e5b28398e: Log is configured to *not* fsync() on all Append() calls
I20260925 04:42:55.087363 19524 log.cc:824] T a081b7f33c36406da49440781a46e7dd P a1b0fa80719745d7b975b3ba1d5449de: Log is configured to *not* fsync() on all Append() calls
I20260925 04:42:55.087358 19525 log.cc:824] T a081b7f33c36406da49440781a46e7dd P d2299066cb27477e9bd5ef207ff9a1dc: Log is configured to *not* fsync() on all Append() calls
I20260925 04:42:55.089104 19524 tablet_bootstrap.cc:492] T a081b7f33c36406da49440781a46e7dd P a1b0fa80719745d7b975b3ba1d5449de: No bootstrap required, opened a new log
I20260925 04:42:55.089155 19523 tablet_bootstrap.cc:492] T a081b7f33c36406da49440781a46e7dd P dffa177b4cb94f03b151246e5b28398e: No bootstrap required, opened a new log
I20260925 04:42:55.089162 19525 tablet_bootstrap.cc:492] T a081b7f33c36406da49440781a46e7dd P d2299066cb27477e9bd5ef207ff9a1dc: No bootstrap required, opened a new log
I20260925 04:42:55.089318 19524 ts_tablet_manager.cc:1397] T a081b7f33c36406da49440781a46e7dd P a1b0fa80719745d7b975b3ba1d5449de: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 04:42:55.089329 19523 ts_tablet_manager.cc:1397] T a081b7f33c36406da49440781a46e7dd P dffa177b4cb94f03b151246e5b28398e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 04:42:55.089401 19525 ts_tablet_manager.cc:1397] T a081b7f33c36406da49440781a46e7dd P d2299066cb27477e9bd5ef207ff9a1dc: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 04:42:55.093056 19524 raft_consensus.cc:348] T a081b7f33c36406da49440781a46e7dd P a1b0fa80719745d7b975b3ba1d5449de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2299066cb27477e9bd5ef207ff9a1dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41571 } } peers { permanent_uuid: "dffa177b4cb94f03b151246e5b28398e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34587 } } peers { permanent_uuid: "a1b0fa80719745d7b975b3ba1d5449de" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46295 } }
I20260925 04:42:55.093313 19524 raft_consensus.cc:374] T a081b7f33c36406da49440781a46e7dd P a1b0fa80719745d7b975b3ba1d5449de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:42:55.093372 19524 raft_consensus.cc:729] T a081b7f33c36406da49440781a46e7dd P a1b0fa80719745d7b975b3ba1d5449de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1b0fa80719745d7b975b3ba1d5449de, State: Initialized, Role: FOLLOWER
I20260925 04:42:55.093680 19523 raft_consensus.cc:348] T a081b7f33c36406da49440781a46e7dd P dffa177b4cb94f03b151246e5b28398e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2299066cb27477e9bd5ef207ff9a1dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41571 } } peers { permanent_uuid: "dffa177b4cb94f03b151246e5b28398e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34587 } } peers { permanent_uuid: "a1b0fa80719745d7b975b3ba1d5449de" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46295 } }
I20260925 04:42:55.093680 19525 raft_consensus.cc:348] T a081b7f33c36406da49440781a46e7dd P d2299066cb27477e9bd5ef207ff9a1dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2299066cb27477e9bd5ef207ff9a1dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41571 } } peers { permanent_uuid: "dffa177b4cb94f03b151246e5b28398e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34587 } } peers { permanent_uuid: "a1b0fa80719745d7b975b3ba1d5449de" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46295 } }
I20260925 04:42:55.093827 19524 consensus_queue.cc:260] T a081b7f33c36406da49440781a46e7dd P a1b0fa80719745d7b975b3ba1d5449de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2299066cb27477e9bd5ef207ff9a1dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41571 } } peers { permanent_uuid: "dffa177b4cb94f03b151246e5b28398e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34587 } } peers { permanent_uuid: "a1b0fa80719745d7b975b3ba1d5449de" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46295 } }
I20260925 04:42:55.093951 19523 raft_consensus.cc:374] T a081b7f33c36406da49440781a46e7dd P dffa177b4cb94f03b151246e5b28398e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:42:55.093951 19525 raft_consensus.cc:374] T a081b7f33c36406da49440781a46e7dd P d2299066cb27477e9bd5ef207ff9a1dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:42:55.094014 19523 raft_consensus.cc:729] T a081b7f33c36406da49440781a46e7dd P dffa177b4cb94f03b151246e5b28398e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dffa177b4cb94f03b151246e5b28398e, State: Initialized, Role: FOLLOWER
I20260925 04:42:55.094014 19525 raft_consensus.cc:729] T a081b7f33c36406da49440781a46e7dd P d2299066cb27477e9bd5ef207ff9a1dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2299066cb27477e9bd5ef207ff9a1dc, State: Initialized, Role: FOLLOWER
I20260925 04:42:55.094498 19525 consensus_queue.cc:260] T a081b7f33c36406da49440781a46e7dd P d2299066cb27477e9bd5ef207ff9a1dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2299066cb27477e9bd5ef207ff9a1dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41571 } } peers { permanent_uuid: "dffa177b4cb94f03b151246e5b28398e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34587 } } peers { permanent_uuid: "a1b0fa80719745d7b975b3ba1d5449de" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46295 } }
I20260925 04:42:55.094698 19375 heartbeater.cc:503] Master 127.0.98.62:42037 was elected leader, sending a full tablet report...
I20260925 04:42:55.094925 19524 ts_tablet_manager.cc:1428] T a081b7f33c36406da49440781a46e7dd P a1b0fa80719745d7b975b3ba1d5449de: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 04:42:55.095191 19244 heartbeater.cc:503] Master 127.0.98.62:42037 was elected leader, sending a full tablet report...
I20260925 04:42:55.094498 19523 consensus_queue.cc:260] T a081b7f33c36406da49440781a46e7dd P dffa177b4cb94f03b151246e5b28398e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2299066cb27477e9bd5ef207ff9a1dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41571 } } peers { permanent_uuid: "dffa177b4cb94f03b151246e5b28398e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34587 } } peers { permanent_uuid: "a1b0fa80719745d7b975b3ba1d5449de" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46295 } }
I20260925 04:42:55.096494 19525 ts_tablet_manager.cc:1428] T a081b7f33c36406da49440781a46e7dd P d2299066cb27477e9bd5ef207ff9a1dc: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260925 04:42:55.096959 19523 ts_tablet_manager.cc:1428] T a081b7f33c36406da49440781a46e7dd P dffa177b4cb94f03b151246e5b28398e: Time spent starting tablet: real 0.008s user 0.000s sys 0.006s
I20260925 04:42:55.097285 19506 heartbeater.cc:503] Master 127.0.98.62:42037 was elected leader, sending a full tablet report...
I20260925 04:42:55.102030 19529 raft_consensus.cc:482] T a081b7f33c36406da49440781a46e7dd P a1b0fa80719745d7b975b3ba1d5449de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:42:55.102205 19529 raft_consensus.cc:504] T a081b7f33c36406da49440781a46e7dd P a1b0fa80719745d7b975b3ba1d5449de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2299066cb27477e9bd5ef207ff9a1dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41571 } } peers { permanent_uuid: "dffa177b4cb94f03b151246e5b28398e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34587 } } peers { permanent_uuid: "a1b0fa80719745d7b975b3ba1d5449de" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46295 } }
I20260925 04:42:55.103291 19529 leader_election.cc:288] T a081b7f33c36406da49440781a46e7dd P a1b0fa80719745d7b975b3ba1d5449de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2299066cb27477e9bd5ef207ff9a1dc (127.0.98.1:41571), dffa177b4cb94f03b151246e5b28398e (127.0.98.3:34587)
I20260925 04:42:55.106945 19377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:55.103488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41571 (local address 127.0.98.2:56569)
0925 04:42:55.103783 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:55.103798 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:42:55.103896 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:42:55.104113 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:42:55.104115 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:42:55.104129 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:42:55.104241 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:55.104249 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:55.104921 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:55.104924 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:55.105899 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:55.105909 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:55.106565 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:55.106569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:55.106666 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:55.106690 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 04:42:55.106757 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 04:42:55.106821 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191}
I20260925 04:42:55.107089 19520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:55.103632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58079 (local address 127.0.98.3:34587)
0925 04:42:55.103736 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:55.103740 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:55.103956 (+ 216us) server_negotiation.cc:407] Connection header received
0925 04:42:55.104056 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:55.104059 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:55.104099 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:55.104156 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:42:55.104386 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:55.104902 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:55.106154 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:55.106573 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:55.106668 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:55.106732 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:55.106916 (+ 184us) server_negotiation.cc:299] Negotiation successful
0925 04:42:55.106984 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 04:42:55.107103 19532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:55.103521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34587 (local address 127.0.98.2:58079)
0925 04:42:55.103875 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:55.103886 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 04:42:55.103977 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:42:55.104171 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:42:55.104174 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:42:55.104188 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:42:55.104295 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:55.104300 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:55.105003 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:55.105006 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:55.106042 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:55.106052 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:55.106699 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:55.106703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:55.106792 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:55.106824 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 04:42:55.106895 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 04:42:55.106962 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":159,"threads_started":1}
I20260925 04:42:55.107308 19518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:55.103544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56569 (local address 127.0.98.1:41571)
0925 04:42:55.103682 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:55.103687 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:55.103865 (+ 178us) server_negotiation.cc:407] Connection header received
0925 04:42:55.103985 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:55.103989 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:55.104039 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:55.104096 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:42:55.104341 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:55.104827 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:55.106030 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:55.106457 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:55.106541 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:55.106590 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:55.107107 (+ 517us) server_negotiation.cc:299] Negotiation successful
0925 04:42:55.107185 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 04:42:55.107672 19460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a081b7f33c36406da49440781a46e7dd" candidate_uuid: "a1b0fa80719745d7b975b3ba1d5449de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dffa177b4cb94f03b151246e5b28398e" is_pre_election: true
I20260925 04:42:55.107933 19198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a081b7f33c36406da49440781a46e7dd" candidate_uuid: "a1b0fa80719745d7b975b3ba1d5449de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2299066cb27477e9bd5ef207ff9a1dc" is_pre_election: true
I20260925 04:42:55.108013 19460 raft_consensus.cc:2454] T a081b7f33c36406da49440781a46e7dd P dffa177b4cb94f03b151246e5b28398e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1b0fa80719745d7b975b3ba1d5449de in term 0.
I20260925 04:42:55.108202 19198 raft_consensus.cc:2454] T a081b7f33c36406da49440781a46e7dd P d2299066cb27477e9bd5ef207ff9a1dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1b0fa80719745d7b975b3ba1d5449de in term 0.
I20260925 04:42:55.108570 19265 leader_election.cc:302] T a081b7f33c36406da49440781a46e7dd P a1b0fa80719745d7b975b3ba1d5449de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1b0fa80719745d7b975b3ba1d5449de, dffa177b4cb94f03b151246e5b28398e; no voters:
I20260925 04:42:55.108856 19529 raft_consensus.cc:2781] T a081b7f33c36406da49440781a46e7dd P a1b0fa80719745d7b975b3ba1d5449de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:42:55.108937 19529 raft_consensus.cc:482] T a081b7f33c36406da49440781a46e7dd P a1b0fa80719745d7b975b3ba1d5449de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:42:55.108974 19529 raft_consensus.cc:3037] T a081b7f33c36406da49440781a46e7dd P a1b0fa80719745d7b975b3ba1d5449de [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:42:55.109941 19529 raft_consensus.cc:504] T a081b7f33c36406da49440781a46e7dd P a1b0fa80719745d7b975b3ba1d5449de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2299066cb27477e9bd5ef207ff9a1dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41571 } } peers { permanent_uuid: "dffa177b4cb94f03b151246e5b28398e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34587 } } peers { permanent_uuid: "a1b0fa80719745d7b975b3ba1d5449de" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46295 } }
I20260925 04:42:55.110245 19529 leader_election.cc:288] T a081b7f33c36406da49440781a46e7dd P a1b0fa80719745d7b975b3ba1d5449de [CANDIDATE]: Term 1 election: Requested vote from peers d2299066cb27477e9bd5ef207ff9a1dc (127.0.98.1:41571), dffa177b4cb94f03b151246e5b28398e (127.0.98.3:34587)
I20260925 04:42:55.110642 19198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a081b7f33c36406da49440781a46e7dd" candidate_uuid: "a1b0fa80719745d7b975b3ba1d5449de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2299066cb27477e9bd5ef207ff9a1dc"
I20260925 04:42:55.110765 19460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a081b7f33c36406da49440781a46e7dd" candidate_uuid: "a1b0fa80719745d7b975b3ba1d5449de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dffa177b4cb94f03b151246e5b28398e"
I20260925 04:42:55.110760 19198 raft_consensus.cc:3037] T a081b7f33c36406da49440781a46e7dd P d2299066cb27477e9bd5ef207ff9a1dc [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:42:55.110900 19460 raft_consensus.cc:3037] T a081b7f33c36406da49440781a46e7dd P dffa177b4cb94f03b151246e5b28398e [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:42:55.111840 19198 raft_consensus.cc:2454] T a081b7f33c36406da49440781a46e7dd P d2299066cb27477e9bd5ef207ff9a1dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1b0fa80719745d7b975b3ba1d5449de in term 1.
I20260925 04:42:55.112005 19460 raft_consensus.cc:2454] T a081b7f33c36406da49440781a46e7dd P dffa177b4cb94f03b151246e5b28398e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1b0fa80719745d7b975b3ba1d5449de in term 1.
I20260925 04:42:55.112180 19262 leader_election.cc:302] T a081b7f33c36406da49440781a46e7dd P a1b0fa80719745d7b975b3ba1d5449de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1b0fa80719745d7b975b3ba1d5449de, d2299066cb27477e9bd5ef207ff9a1dc; no voters:
I20260925 04:42:55.112341 19529 raft_consensus.cc:2781] T a081b7f33c36406da49440781a46e7dd P a1b0fa80719745d7b975b3ba1d5449de [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:42:55.112532 19529 raft_consensus.cc:686] T a081b7f33c36406da49440781a46e7dd P a1b0fa80719745d7b975b3ba1d5449de [term 1 LEADER]: Becoming Leader. State: Replica: a1b0fa80719745d7b975b3ba1d5449de, State: Running, Role: LEADER
I20260925 04:42:55.112870 19529 consensus_queue.cc:237] T a081b7f33c36406da49440781a46e7dd P a1b0fa80719745d7b975b3ba1d5449de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2299066cb27477e9bd5ef207ff9a1dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41571 } } peers { permanent_uuid: "dffa177b4cb94f03b151246e5b28398e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34587 } } peers { permanent_uuid: "a1b0fa80719745d7b975b3ba1d5449de" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46295 } }
I20260925 04:42:55.116181 19056 catalog_manager.cc:5495] T a081b7f33c36406da49440781a46e7dd P a1b0fa80719745d7b975b3ba1d5449de reported cstate change: term changed from 0 to 1, leader changed from <none> to a1b0fa80719745d7b975b3ba1d5449de (127.0.98.2). New cstate: current_term: 1 leader_uuid: "a1b0fa80719745d7b975b3ba1d5449de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2299066cb27477e9bd5ef207ff9a1dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dffa177b4cb94f03b151246e5b28398e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1b0fa80719745d7b975b3ba1d5449de" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46295 } health_report { overall_health: HEALTHY } } }
W20260925 04:42:55.129062 19245 tablet.cc:2369] T a081b7f33c36406da49440781a46e7dd P d2299066cb27477e9bd5ef207ff9a1dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:42:55.138577 19522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:55.135002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59310 (local address 127.0.98.2:46295)
0925 04:42:55.135129 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:55.135133 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:55.135146 (+ 13us) server_negotiation.cc:407] Connection header received
0925 04:42:55.135231 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:55.135234 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:55.135299 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:55.135386 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:42:55.135587 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:55.136165 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:55.137063 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:55.137283 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:55.137363 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:55.137683 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:42:55.138122 (+ 439us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:42:55.138233 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:55.138359 (+ 126us) server_negotiation.cc:299] Negotiation successful
0925 04:42:55.138396 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 04:42:55.144734 19198 raft_consensus.cc:1261] T a081b7f33c36406da49440781a46e7dd P d2299066cb27477e9bd5ef207ff9a1dc [term 1 FOLLOWER]: Refusing update from remote peer a1b0fa80719745d7b975b3ba1d5449de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:42:55.144743 19460 raft_consensus.cc:1261] T a081b7f33c36406da49440781a46e7dd P dffa177b4cb94f03b151246e5b28398e [term 1 FOLLOWER]: Refusing update from remote peer a1b0fa80719745d7b975b3ba1d5449de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:42:55.145493 19533 consensus_queue.cc:1035] T a081b7f33c36406da49440781a46e7dd P a1b0fa80719745d7b975b3ba1d5449de [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2299066cb27477e9bd5ef207ff9a1dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:42:55.145736 19529 consensus_queue.cc:1035] T a081b7f33c36406da49440781a46e7dd P a1b0fa80719745d7b975b3ba1d5449de [LEADER]: Connected to new peer: Peer: permanent_uuid: "dffa177b4cb94f03b151246e5b28398e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:42:55.153823 19537 mvcc.cc:205] Tried to move back new op lower bound from 7333115392585805824 to 7333115392465784832. Current Snapshot: MvccSnapshot[applied={T|T < 7333115392585805824}]
I20260925 04:42:55.154091 19536 mvcc.cc:205] Tried to move back new op lower bound from 7333115392585805824 to 7333115392465784832. Current Snapshot: MvccSnapshot[applied={T|T < 7333115392585805824}]
I20260925 04:42:55.165503 19055 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:42:55.165683 19055 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:42:55.168808 19055 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e2cb7fe70f244423b604b679c3d5bbbe: Sending DeleteTablet for 3 replicas of tablet a081b7f33c36406da49440781a46e7dd
I20260925 04:42:55.169821 19440 tablet_service.cc:1514] Processing DeleteTablet for tablet a081b7f33c36406da49440781a46e7dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:42:55 UTC) from {username='slave'} at 127.0.0.1:35018
I20260925 04:42:55.169802 19178 tablet_service.cc:1514] Processing DeleteTablet for tablet a081b7f33c36406da49440781a46e7dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:42:55 UTC) from {username='slave'} at 127.0.0.1:42556
I20260925 04:42:55.169900 19309 tablet_service.cc:1514] Processing DeleteTablet for tablet a081b7f33c36406da49440781a46e7dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:42:55 UTC) from {username='slave'} at 127.0.0.1:59308
I20260925 04:42:55.170411 19546 tablet_replica.cc:330] T a081b7f33c36406da49440781a46e7dd P dffa177b4cb94f03b151246e5b28398e: stopping tablet replica
I20260925 04:42:55.170677 19545 tablet_replica.cc:330] T a081b7f33c36406da49440781a46e7dd P d2299066cb27477e9bd5ef207ff9a1dc: stopping tablet replica
I20260925 04:42:55.170979 19546 raft_consensus.cc:2229] T a081b7f33c36406da49440781a46e7dd P dffa177b4cb94f03b151246e5b28398e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:42:55.171343 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 19116
I20260925 04:42:55.171383 19546 raft_consensus.cc:2258] T a081b7f33c36406da49440781a46e7dd P dffa177b4cb94f03b151246e5b28398e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:42:55.171028 19547 tablet_replica.cc:330] T a081b7f33c36406da49440781a46e7dd P a1b0fa80719745d7b975b3ba1d5449de: stopping tablet replica
I20260925 04:42:55.171943 19547 raft_consensus.cc:2229] T a081b7f33c36406da49440781a46e7dd P a1b0fa80719745d7b975b3ba1d5449de [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:42:55.172494 19547 raft_consensus.cc:2258] T a081b7f33c36406da49440781a46e7dd P a1b0fa80719745d7b975b3ba1d5449de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:42:55.174176 19546 ts_tablet_manager.cc:1905] T a081b7f33c36406da49440781a46e7dd P dffa177b4cb94f03b151246e5b28398e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:42:55.176608 19546 ts_tablet_manager.cc:1918] T a081b7f33c36406da49440781a46e7dd P dffa177b4cb94f03b151246e5b28398e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:42:55.176705 19546 log.cc:1196] T a081b7f33c36406da49440781a46e7dd P dffa177b4cb94f03b151246e5b28398e: Deleting WAL directory at /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311123478858-392-0/minicluster-data/ts-2/wal/wals/a081b7f33c36406da49440781a46e7dd
I20260925 04:42:55.176959 19546 ts_tablet_manager.cc:1939] T a081b7f33c36406da49440781a46e7dd P dffa177b4cb94f03b151246e5b28398e: Deleting consensus metadata
I20260925 04:42:55.178113 19547 ts_tablet_manager.cc:1905] T a081b7f33c36406da49440781a46e7dd P a1b0fa80719745d7b975b3ba1d5449de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:42:55.178370 19043 catalog_manager.cc:4841] TS dffa177b4cb94f03b151246e5b28398e (127.0.98.3:34587): tablet a081b7f33c36406da49440781a46e7dd (table test-table [id=d198caf5e51c44d88dcd147de6592094]) successfully deleted
W20260925 04:42:55.180301 19040 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41571 (error 108)
I20260925 04:42:55.180511 19547 ts_tablet_manager.cc:1918] T a081b7f33c36406da49440781a46e7dd P a1b0fa80719745d7b975b3ba1d5449de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:42:55.180606 19547 log.cc:1196] T a081b7f33c36406da49440781a46e7dd P a1b0fa80719745d7b975b3ba1d5449de: Deleting WAL directory at /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311123478858-392-0/minicluster-data/ts-1/wal/wals/a081b7f33c36406da49440781a46e7dd
I20260925 04:42:55.180866 19547 ts_tablet_manager.cc:1939] T a081b7f33c36406da49440781a46e7dd P a1b0fa80719745d7b975b3ba1d5449de: Deleting consensus metadata
I20260925 04:42:55.181156 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 19247
I20260925 04:42:55.181872 19519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:55.181516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41571 (local address 127.0.0.1:42562)
0925 04:42:55.181706 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:55.181779 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
W20260925 04:42:55.182036 19040 catalog_manager.cc:4584] TS d2299066cb27477e9bd5ef207ff9a1dc (127.0.98.1:41571): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a081b7f33c36406da49440781a46e7dd: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41571: connect: Connection refused (error 111)
I20260925 04:42:55.186446 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 19378
I20260925 04:42:55.187532 19519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:55.187226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46295 (local address 127.0.0.1:59316)
0925 04:42:55.187385 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:55.187439 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:46295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260925 04:42:55.191385 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 19025
2026-09-25T04:42:55Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1032 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:42:55 dist-test-slave-dc95 krb5kdc[19551](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:42:55 dist-test-slave-dc95 krb5kdc[19551](info): set up 2 sockets
Sep 25 04:42:55 dist-test-slave-dc95 krb5kdc[19551](info): commencing operation
krb5kdc: starting...
W20260925 04:42:56.147464 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/krb5kdc/test-user.keytab.
Sep 25 04:42:56 dist-test-slave-dc95 krb5kdc[19551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:42:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:42:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 04:42:56.274554 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42211
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42211
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:42:56.385512 19564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:42:56.385835 19564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:42:56.385895 19564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:42:56.390136 19564 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:42:56.390218 19564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:42:56.390245 19564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:42:56.390265 19564 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:42:56.390285 19564 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:42:56.395941 19564 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42211
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:42:56.397154 19564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:42:56.398195 19564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:42:56.406911 19569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:42:56.406965 19564 server_base.cc:1034] running on GCE node
W20260925 04:42:56.406972 19572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:42:56.406932 19570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:42:56.407791 19564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:42:56.409003 19564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:42:56.410171 19564 hybrid_clock.cc:648] HybridClock initialized: now 1790311376410145 us; error 43 us; skew 500 ppm
Sep 25 04:42:56 dist-test-slave-dc95 krb5kdc[19551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311376, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:42:56.413278 19564 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 04:42:56.414376 19564 webserver.cc:460] Webserver started at http://127.0.98.62:45577/ using document root <none> and password file <none>
I20260925 04:42:56.414996 19564 fs_manager.cc:359] Metadata directory not provided
I20260925 04:42:56.415064 19564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:42:56.415287 19564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:42:56.417456 19564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/minicluster-data/master-0/data/instance:
uuid: "57f55d294c1741ec8af839df0a26c033"
format_stamp: "Formatted at 2026-09-25 04:42:56 on dist-test-slave-dc95"
server_key: "e829e81c1226dcd1c5a7b012ac3ac15a"
server_key_iv: "a9157450ad55404281f6b1f613e6412b"
server_key_version: "encryptionkey@0"
I20260925 04:42:56.418015 19564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/minicluster-data/master-0/wal/instance:
uuid: "57f55d294c1741ec8af839df0a26c033"
format_stamp: "Formatted at 2026-09-25 04:42:56 on dist-test-slave-dc95"
server_key: "e829e81c1226dcd1c5a7b012ac3ac15a"
server_key_iv: "a9157450ad55404281f6b1f613e6412b"
server_key_version: "encryptionkey@0"
I20260925 04:42:56.421942 19564 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 04:42:56.424810 19579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:56.426190 19564 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:42:56.426329 19564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/minicluster-data/master-0/wal
uuid: "57f55d294c1741ec8af839df0a26c033"
format_stamp: "Formatted at 2026-09-25 04:42:56 on dist-test-slave-dc95"
server_key: "e829e81c1226dcd1c5a7b012ac3ac15a"
server_key_iv: "a9157450ad55404281f6b1f613e6412b"
server_key_version: "encryptionkey@0"
I20260925 04:42:56.426455 19564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:42:56.442466 19564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:42:56.445771 19564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:42:56.445955 19564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:42:56.453171 19564 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42211
I20260925 04:42:56.453244 19631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42211 every 8 connection(s)
I20260925 04:42:56.454154 19564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
I20260925 04:42:56.457026 19632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:42:56.461326 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 19564
I20260925 04:42:56.461463 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/minicluster-data/master-0/wal/instance
I20260925 04:42:56.461714 392 external_mini_cluster.cc:1428] Setting key c203c236380cf6fbef8d9a388610eb70
I20260925 04:42:56.463091 19632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57f55d294c1741ec8af839df0a26c033: Bootstrap starting.
I20260925 04:42:56.465556 19632 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57f55d294c1741ec8af839df0a26c033: Neither blocks nor log segments found. Creating new log.
I20260925 04:42:56.466391 19632 log.cc:824] T 00000000000000000000000000000000 P 57f55d294c1741ec8af839df0a26c033: Log is configured to *not* fsync() on all Append() calls
I20260925 04:42:56.468750 19632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57f55d294c1741ec8af839df0a26c033: No bootstrap required, opened a new log
Sep 25 04:42:56 dist-test-slave-dc95 krb5kdc[19551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:42:56.473042 19632 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57f55d294c1741ec8af839df0a26c033 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57f55d294c1741ec8af839df0a26c033" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42211 } }
I20260925 04:42:56.473281 19632 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57f55d294c1741ec8af839df0a26c033 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:42:56.473315 19632 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57f55d294c1741ec8af839df0a26c033 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57f55d294c1741ec8af839df0a26c033, State: Initialized, Role: FOLLOWER
I20260925 04:42:56.473819 19632 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57f55d294c1741ec8af839df0a26c033 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57f55d294c1741ec8af839df0a26c033" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42211 } }
I20260925 04:42:56.473982 19632 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57f55d294c1741ec8af839df0a26c033 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:42:56.474052 19632 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57f55d294c1741ec8af839df0a26c033 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:42:56.474148 19632 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57f55d294c1741ec8af839df0a26c033 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:42:56.475046 19632 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57f55d294c1741ec8af839df0a26c033 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57f55d294c1741ec8af839df0a26c033" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42211 } }
I20260925 04:42:56.475404 19632 leader_election.cc:302] T 00000000000000000000000000000000 P 57f55d294c1741ec8af839df0a26c033 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57f55d294c1741ec8af839df0a26c033; no voters:
I20260925 04:42:56.475780 19632 leader_election.cc:288] T 00000000000000000000000000000000 P 57f55d294c1741ec8af839df0a26c033 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:42:56.475867 19637 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57f55d294c1741ec8af839df0a26c033 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:42:56.476056 19637 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57f55d294c1741ec8af839df0a26c033 [term 1 LEADER]: Becoming Leader. State: Replica: 57f55d294c1741ec8af839df0a26c033, State: Running, Role: LEADER
I20260925 04:42:56.476424 19637 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57f55d294c1741ec8af839df0a26c033 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57f55d294c1741ec8af839df0a26c033" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42211 } }
I20260925 04:42:56.476547 19635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:56.463146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35426 (local address 127.0.98.62:42211)
0925 04:42:56.463590 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:56.463627 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:56.463671 (+ 44us) server_negotiation.cc:407] Connection header received
0925 04:42:56.464369 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:56.464387 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:56.464729 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:56.465012 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:42:56.465898 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:56.467042 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:56.468261 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:56.468747 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:56.468907 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:56.471683 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:42:56.471688 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:42:56.471703 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:42:56.471734 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:42:56.473474 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:42:56.473923 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:42:56.473928 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:42:56.473932 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:42:56.474091 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:42:56.474364 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:42:56.474367 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:42:56.474369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:42:56.474764 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:42:56.474946 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:56.475232 (+ 286us) server_negotiation.cc:299] Negotiation successful
0925 04:42:56.475522 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":138,"threads_started":1}
I20260925 04:42:56.477300 19632 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57f55d294c1741ec8af839df0a26c033 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:42:56.479456 19639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57f55d294c1741ec8af839df0a26c033 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57f55d294c1741ec8af839df0a26c033. Latest consensus state: current_term: 1 leader_uuid: "57f55d294c1741ec8af839df0a26c033" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57f55d294c1741ec8af839df0a26c033" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42211 } } }
I20260925 04:42:56.479621 19639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57f55d294c1741ec8af839df0a26c033 [sys.catalog]: This master's current role is: LEADER
I20260925 04:42:56.479998 19653 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:42:56.480003 19638 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57f55d294c1741ec8af839df0a26c033 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57f55d294c1741ec8af839df0a26c033" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57f55d294c1741ec8af839df0a26c033" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42211 } } }
I20260925 04:42:56.480160 19638 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57f55d294c1741ec8af839df0a26c033 [sys.catalog]: This master's current role is: LEADER
I20260925 04:42:56.482846 19653 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:42:56.489101 19653 catalog_manager.cc:1347] Generated new cluster ID: 4f0d63be13594f079b259c7419d4877b
I20260925 04:42:56.489188 19653 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:42:56.495986 19653 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:42:56.497069 19653 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:42:56.505141 19653 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57f55d294c1741ec8af839df0a26c033: Generated new TSK 0
I20260925 04:42:56.506026 19653 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 04:42:56.543558 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42211
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:42:56.655731 19659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:42:56.656032 19659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:42:56.656091 19659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:42:56.660320 19659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:42:56.660399 19659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:42:56.660490 19659 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 04:42:56.666163 19659 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19659
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:42:56.667344 19659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:42:56.668596 19659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:42:56.678934 19664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:42:56.679010 19667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:42:56.678972 19665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:42:56.679344 19659 server_base.cc:1034] running on GCE node
I20260925 04:42:56.679867 19659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:42:56.680572 19659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:42:56.681738 19659 hybrid_clock.cc:648] HybridClock initialized: now 1790311376681720 us; error 36 us; skew 500 ppm
Sep 25 04:42:56 dist-test-slave-dc95 krb5kdc[19551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311376, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:42:56.684578 19659 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 04:42:56.685513 19659 webserver.cc:460] Webserver started at http://127.0.98.1:43817/ using document root <none> and password file <none>
I20260925 04:42:56.686100 19659 fs_manager.cc:359] Metadata directory not provided
I20260925 04:42:56.686168 19659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:42:56.686371 19659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:42:56.688566 19659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/minicluster-data/ts-0/data/instance:
uuid: "22191865f6ca4fb191c9f9ff5f64547d"
format_stamp: "Formatted at 2026-09-25 04:42:56 on dist-test-slave-dc95"
server_key: "9cbdc662a8a4390dc8c87df0d8f1adab"
server_key_iv: "2efcfef8e5ddb7c42173c55d2d74398d"
server_key_version: "encryptionkey@0"
I20260925 04:42:56.689126 19659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/minicluster-data/ts-0/wal/instance:
uuid: "22191865f6ca4fb191c9f9ff5f64547d"
format_stamp: "Formatted at 2026-09-25 04:42:56 on dist-test-slave-dc95"
server_key: "9cbdc662a8a4390dc8c87df0d8f1adab"
server_key_iv: "2efcfef8e5ddb7c42173c55d2d74398d"
server_key_version: "encryptionkey@0"
I20260925 04:42:56.693073 19659 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:42:56.695762 19674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:56.697288 19659 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:42:56.697463 19659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/minicluster-data/ts-0/wal
uuid: "22191865f6ca4fb191c9f9ff5f64547d"
format_stamp: "Formatted at 2026-09-25 04:42:56 on dist-test-slave-dc95"
server_key: "9cbdc662a8a4390dc8c87df0d8f1adab"
server_key_iv: "2efcfef8e5ddb7c42173c55d2d74398d"
server_key_version: "encryptionkey@0"
I20260925 04:42:56.697587 19659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:42:56.717537 19659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:42:56.720956 19659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:42:56.721153 19659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:42:56.721906 19659 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:42:56.722811 19659 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:42:56.722862 19659 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:56.722916 19659 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:42:56.722942 19659 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:56.732093 19659 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44595
I20260925 04:42:56.732116 19787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44595 every 8 connection(s)
I20260925 04:42:56.733011 19659 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
I20260925 04:42:56.740563 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 19659
I20260925 04:42:56.740710 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/minicluster-data/ts-0/wal/instance
I20260925 04:42:56.741036 392 external_mini_cluster.cc:1428] Setting key b697ec48828e1327e2e257daf2db8781
Sep 25 04:42:56 dist-test-slave-dc95 krb5kdc[19551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311376, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:42:56.747090 19635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:56.734768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54897 (local address 127.0.98.62:42211)
0925 04:42:56.734896 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:56.734899 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:56.735818 (+ 919us) server_negotiation.cc:407] Connection header received
0925 04:42:56.737129 (+ 1311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:56.737132 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:56.737182 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:56.737263 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:42:56.738474 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:56.739053 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:56.740438 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:56.740791 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:56.740909 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:56.743412 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:42:56.743416 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:42:56.743418 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:42:56.743446 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:42:56.744988 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:42:56.745499 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:42:56.745503 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:42:56.745504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:42:56.745590 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:42:56.745890 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:42:56.745893 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:42:56.745895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:42:56.746125 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:42:56.746225 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:56.746760 (+ 535us) server_negotiation.cc:299] Negotiation successful
0925 04:42:56.746915 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 04:42:56.747869 19790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:56.735156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42211 (local address 127.0.98.1:54897)
0925 04:42:56.735655 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:56.735712 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 04:42:56.736876 (+ 1164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:42:56.737426 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:42:56.737434 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:42:56.737800 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:42:56.738323 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:56.738336 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:56.739207 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:56.739210 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:56.740248 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:56.740258 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:56.740950 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:56.740953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:56.741144 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:56.741699 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:42:56.741715 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:42:56.743234 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:42:56.745103 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:42:56.745109 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:42:56.745122 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:42:56.745369 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:42:56.745673 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:42:56.745676 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:42:56.745677 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:42:56.745811 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:42:56.746231 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:42:56.746237 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:42:56.746528 (+ 291us) client_negotiation.cc:769] Sending connection context
0925 04:42:56.746689 (+ 161us) client_negotiation.cc:240] Negotiation successful
0925 04:42:56.746959 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":125,"threads_started":1}
I20260925 04:42:56.749058 19788 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42211
I20260925 04:42:56.749325 19788 heartbeater.cc:464] Registering TS with master...
I20260925 04:42:56.749943 19788 heartbeater.cc:511] Master 127.0.98.62:42211 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 04:42:56.751540 19596 ts_manager.cc:194] Registered new tserver with Master: 22191865f6ca4fb191c9f9ff5f64547d (127.0.98.1:44595)
I20260925 04:42:56.752746 19596 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54897
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 04:42:56.768993 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42211
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:42:56.880730 19793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:42:56.881019 19793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:42:56.881069 19793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:42:56.885258 19793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:42:56.885329 19793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:42:56.885402 19793 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 04:42:56.890901 19793 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:42:56.892097 19793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:42:56.893148 19793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:42:56.902828 19798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:42:56.902873 19799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:42:56.903014 19801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:42:56.903050 19793 server_base.cc:1034] running on GCE node
I20260925 04:42:56.903519 19793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:42:56.904191 19793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:42:56.905390 19793 hybrid_clock.cc:648] HybridClock initialized: now 1790311376905370 us; error 33 us; skew 500 ppm
Sep 25 04:42:56 dist-test-slave-dc95 krb5kdc[19551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311376, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:42:56.908140 19793 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 04:42:56.909044 19793 webserver.cc:460] Webserver started at http://127.0.98.2:45979/ using document root <none> and password file <none>
I20260925 04:42:56.909592 19793 fs_manager.cc:359] Metadata directory not provided
I20260925 04:42:56.909658 19793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:42:56.909847 19793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:42:56.911983 19793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/minicluster-data/ts-1/data/instance:
uuid: "e47fa4603d63429c9da8792553b713e0"
format_stamp: "Formatted at 2026-09-25 04:42:56 on dist-test-slave-dc95"
server_key: "0de0c5922e60658eab62a0a94b0902e7"
server_key_iv: "6b70c6dcd60d5e85a68590a18509b09c"
server_key_version: "encryptionkey@0"
I20260925 04:42:56.912537 19793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e47fa4603d63429c9da8792553b713e0"
format_stamp: "Formatted at 2026-09-25 04:42:56 on dist-test-slave-dc95"
server_key: "0de0c5922e60658eab62a0a94b0902e7"
server_key_iv: "6b70c6dcd60d5e85a68590a18509b09c"
server_key_version: "encryptionkey@0"
I20260925 04:42:56.916400 19793 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 04:42:56.919020 19808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:56.920392 19793 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:42:56.920511 19793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/minicluster-data/ts-1/wal
uuid: "e47fa4603d63429c9da8792553b713e0"
format_stamp: "Formatted at 2026-09-25 04:42:56 on dist-test-slave-dc95"
server_key: "0de0c5922e60658eab62a0a94b0902e7"
server_key_iv: "6b70c6dcd60d5e85a68590a18509b09c"
server_key_version: "encryptionkey@0"
I20260925 04:42:56.920616 19793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:42:56.938115 19793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:42:56.940948 19793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:42:56.941120 19793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:42:56.941682 19793 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:42:56.942469 19793 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:42:56.942515 19793 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:56.942567 19793 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:42:56.942593 19793 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:56.950824 19793 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44531
I20260925 04:42:56.950845 19921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44531 every 8 connection(s)
I20260925 04:42:56.951812 19793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
I20260925 04:42:56.955633 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 19793
I20260925 04:42:56.955780 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/minicluster-data/ts-1/wal/instance
I20260925 04:42:56.956111 392 external_mini_cluster.cc:1428] Setting key 27caefb8044a4fa481488a83612328cd
Sep 25 04:42:56 dist-test-slave-dc95 krb5kdc[19551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311376, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260925 04:42:56.966199 19635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:56.953724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35557 (local address 127.0.98.62:42211)
0925 04:42:56.953898 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:56.953903 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:56.954754 (+ 851us) server_negotiation.cc:407] Connection header received
0925 04:42:56.956249 (+ 1495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:56.956253 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:56.956304 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:56.956384 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:42:56.957564 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:56.958071 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:56.959454 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:56.959806 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:56.959917 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:56.962464 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:42:56.962468 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:42:56.962470 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:42:56.962495 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:42:56.963950 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:42:56.964551 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:42:56.964554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:42:56.964555 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:42:56.964641 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:42:56.965055 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:42:56.965058 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:42:56.965059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:42:56.965315 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:42:56.965404 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:56.965908 (+ 504us) server_negotiation.cc:299] Negotiation successful
0925 04:42:56.966032 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 04:42:56.967024 19924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:56.954093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42211 (local address 127.0.98.2:35557)
0925 04:42:56.954602 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:56.954645 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 04:42:56.956012 (+ 1367us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:42:56.956584 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:42:56.956591 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:42:56.956938 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:42:56.957422 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:56.957440 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:56.958195 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:56.958200 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:56.959338 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:56.959347 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:56.959956 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:56.959960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:56.960097 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:56.960642 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:42:56.960658 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:42:56.962232 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:42:56.964070 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:42:56.964077 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:42:56.964094 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:42:56.964438 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:42:56.964727 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:42:56.964731 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:42:56.964733 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:42:56.964915 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:42:56.965409 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:42:56.965414 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:42:56.965669 (+ 255us) client_negotiation.cc:769] Sending connection context
0925 04:42:56.965849 (+ 180us) client_negotiation.cc:240] Negotiation successful
0925 04:42:56.966208 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":126,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260925 04:42:56.968273 19922 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42211
I20260925 04:42:56.968521 19922 heartbeater.cc:464] Registering TS with master...
I20260925 04:42:56.969015 19922 heartbeater.cc:511] Master 127.0.98.62:42211 requested a full tablet report, sending...
I20260925 04:42:56.970217 19596 ts_manager.cc:194] Registered new tserver with Master: e47fa4603d63429c9da8792553b713e0 (127.0.98.2:44531)
I20260925 04:42:56.970785 19596 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35557
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 04:42:56.983170 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42211
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:42:57.093400 19927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:42:57.093689 19927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:42:57.093750 19927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:42:57.097919 19927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:42:57.097995 19927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:42:57.098081 19927 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 04:42:57.103897 19927 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19927
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:42:57.104964 19927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:42:57.105973 19927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:42:57.115664 19932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:42:57.115597 19933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:42:57.115655 19927 server_base.cc:1034] running on GCE node
W20260925 04:42:57.115655 19935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:42:57.116295 19927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:42:57.116940 19927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:42:57.118120 19927 hybrid_clock.cc:648] HybridClock initialized: now 1790311377118106 us; error 27 us; skew 500 ppm
Sep 25 04:42:57 dist-test-slave-dc95 krb5kdc[19551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311377, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:42:57.120980 19927 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 04:42:57.121946 19927 webserver.cc:460] Webserver started at http://127.0.98.3:41775/ using document root <none> and password file <none>
I20260925 04:42:57.122519 19927 fs_manager.cc:359] Metadata directory not provided
I20260925 04:42:57.122586 19927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:42:57.122786 19927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:42:57.125010 19927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/minicluster-data/ts-2/data/instance:
uuid: "0cd6c4a2df1441d198a5e14539090b8b"
format_stamp: "Formatted at 2026-09-25 04:42:57 on dist-test-slave-dc95"
server_key: "654de289a334d8bcc737b921ac45294d"
server_key_iv: "0fb6074590ad3f25e6ad17f2a2d221b9"
server_key_version: "encryptionkey@0"
I20260925 04:42:57.125566 19927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0cd6c4a2df1441d198a5e14539090b8b"
format_stamp: "Formatted at 2026-09-25 04:42:57 on dist-test-slave-dc95"
server_key: "654de289a334d8bcc737b921ac45294d"
server_key_iv: "0fb6074590ad3f25e6ad17f2a2d221b9"
server_key_version: "encryptionkey@0"
I20260925 04:42:57.129549 19927 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 04:42:57.132258 19942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:57.133558 19927 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:42:57.133678 19927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/minicluster-data/ts-2/wal
uuid: "0cd6c4a2df1441d198a5e14539090b8b"
format_stamp: "Formatted at 2026-09-25 04:42:57 on dist-test-slave-dc95"
server_key: "654de289a334d8bcc737b921ac45294d"
server_key_iv: "0fb6074590ad3f25e6ad17f2a2d221b9"
server_key_version: "encryptionkey@0"
I20260925 04:42:57.133791 19927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:42:57.160080 19927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:42:57.163074 19927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:42:57.163267 19927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:42:57.163908 19927 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:42:57.164767 19927 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:42:57.164814 19927 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:57.164867 19927 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:42:57.164897 19927 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:57.173496 19927 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40999
I20260925 04:42:57.173530 20055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40999 every 8 connection(s)
I20260925 04:42:57.174439 19927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
I20260925 04:42:57.179662 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 19927
I20260925 04:42:57.179791 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/minicluster-data/ts-2/wal/instance
I20260925 04:42:57.180070 392 external_mini_cluster.cc:1428] Setting key 4f67c8a3891ef296ed1d930b866f0367
Sep 25 04:42:57 dist-test-slave-dc95 krb5kdc[19551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311377, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:42:57.189064 19635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:57.176207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39463 (local address 127.0.98.62:42211)
0925 04:42:57.176341 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:57.176345 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:57.177251 (+ 906us) server_negotiation.cc:407] Connection header received
0925 04:42:57.178654 (+ 1403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:57.178658 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:57.178717 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:57.178788 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:42:57.180169 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:57.181001 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:57.182501 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:57.182748 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:57.182840 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:57.185515 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:42:57.185518 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:42:57.185520 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:42:57.185548 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:42:57.186965 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:42:57.187492 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:42:57.187495 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:42:57.187497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:42:57.187576 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:42:57.187930 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:42:57.187934 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:42:57.187935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:42:57.188128 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:42:57.188215 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:57.188760 (+ 545us) server_negotiation.cc:299] Negotiation successful
0925 04:42:57.188896 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 04:42:57.189937 20058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:57.176601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42211 (local address 127.0.98.3:39463)
0925 04:42:57.177099 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:57.177145 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 04:42:57.178449 (+ 1304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:42:57.178974 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:42:57.178980 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:42:57.179410 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:42:57.180013 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:57.180028 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:57.181160 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:57.181164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:57.182354 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:57.182368 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:57.182887 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:57.182890 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:57.183045 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:57.183746 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:42:57.183764 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:42:57.185313 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:42:57.187087 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:42:57.187094 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:42:57.187112 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:42:57.187389 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:42:57.187680 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:42:57.187683 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:42:57.187685 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:42:57.187840 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:42:57.188240 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:42:57.188247 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:42:57.188513 (+ 266us) client_negotiation.cc:769] Sending connection context
0925 04:42:57.188692 (+ 179us) client_negotiation.cc:240] Negotiation successful
0925 04:42:57.189047 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":130,"threads_started":1}
I20260925 04:42:57.191108 20056 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42211
I20260925 04:42:57.191399 20056 heartbeater.cc:464] Registering TS with master...
I20260925 04:42:57.191931 20056 heartbeater.cc:511] Master 127.0.98.62:42211 requested a full tablet report, sending...
I20260925 04:42:57.193038 19596 ts_manager.cc:194] Registered new tserver with Master: 0cd6c4a2df1441d198a5e14539090b8b (127.0.98.3:40999)
I20260925 04:42:57.193773 19596 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39463
I20260925 04:42:57.194228 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:42:57.203114 19635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:57.196087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35442 (local address 127.0.98.62:42211)
0925 04:42:57.196212 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:57.196217 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:57.196388 (+ 171us) server_negotiation.cc:407] Connection header received
0925 04:42:57.196478 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:57.196480 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:57.196516 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:57.196582 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:42:57.197225 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:57.197754 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:57.198662 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:57.198906 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:57.198994 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:57.200329 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:42:57.200332 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:42:57.200334 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:42:57.200360 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:42:57.201648 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:42:57.202071 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:42:57.202074 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:42:57.202075 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:42:57.202150 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:42:57.202380 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:42:57.202381 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:42:57.202382 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:42:57.202587 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:42:57.202712 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:57.202894 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 04:42:57.202988 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 04:42:57.206027 19596 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35442:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:42:57.209218 19596 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:42:57.224540 20071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:57.218749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55634 (local address 127.0.98.2:44531)
0925 04:42:57.219572 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:57.219584 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:57.219627 (+ 43us) server_negotiation.cc:407] Connection header received
0925 04:42:57.219700 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:57.219705 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:57.219892 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:57.220300 (+ 408us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:42:57.220447 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:57.221534 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:57.222972 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:57.223832 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:57.223932 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:57.224071 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:57.224279 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 04:42:57.224384 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":729,"thread_start_us":148,"threads_started":1}
I20260925 04:42:57.224536 20068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:57.218236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44531 (local address 127.0.0.1:55634)
0925 04:42:57.219132 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:57.219143 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 04:42:57.219215 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:42:57.220027 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:42:57.220030 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:42:57.220092 (+ 62us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:42:57.220163 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:57.220168 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:57.221674 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:57.221677 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:57.222851 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:57.222858 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:57.223959 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:57.223963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:57.224078 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:57.224117 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 04:42:57.224234 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 04:42:57.224364 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":842,"mutex_wait_us":17,"thread_start_us":84,"threads_started":1}
I20260925 04:42:57.224857 20069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:57.218056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50040 (local address 127.0.98.1:44595)
0925 04:42:57.218454 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:57.218470 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:57.218820 (+ 350us) server_negotiation.cc:407] Connection header received
0925 04:42:57.219108 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:57.219114 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:57.219344 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:57.219472 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:42:57.220011 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:57.221171 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:57.222958 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:57.223832 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:57.223917 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:57.224071 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:57.224611 (+ 540us) server_negotiation.cc:299] Negotiation successful
0925 04:42:57.224716 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":76,"threads_started":1}
I20260925 04:42:57.225059 20070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:57.218007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40999 (local address 127.0.0.1:58904)
0925 04:42:57.218835 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:57.218848 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:42:57.218933 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:42:57.219870 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:42:57.219873 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:42:57.219890 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:42:57.220221 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:57.220231 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:57.222465 (+ 2234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:57.222469 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:57.223568 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:57.223580 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:57.224680 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:57.224684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:57.224780 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:57.224802 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 04:42:57.224851 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 04:42:57.224914 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":767,"thread_start_us":92,"threads_started":1}
I20260925 04:42:57.224547 20067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:57.217925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44595 (local address 127.0.0.1:50040)
0925 04:42:57.218676 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:57.218728 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 04:42:57.218933 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:42:57.219715 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:42:57.219721 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:42:57.219743 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:42:57.219880 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:57.219887 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:57.221327 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:57.221330 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:57.222818 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:57.222829 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:57.224166 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:57.224169 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:57.224273 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:57.224295 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 04:42:57.224359 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 04:42:57.224427 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"mutex_wait_us":47,"thread_start_us":89,"threads_started":1}
I20260925 04:42:57.225682 20072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:57.219157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58904 (local address 127.0.98.3:40999)
0925 04:42:57.219475 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:57.219485 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:57.219502 (+ 17us) server_negotiation.cc:407] Connection header received
0925 04:42:57.219560 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:57.219563 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:57.219753 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:57.219880 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:42:57.220358 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:57.222307 (+ 1949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:57.223738 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:57.224554 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:57.225253 (+ 699us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:57.225406 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:57.225451 (+ 45us) server_negotiation.cc:299] Negotiation successful
0925 04:42:57.225531 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":119,"threads_started":1}
I20260925 04:42:57.227746 19856 tablet_service.cc:1467] Processing CreateTablet for tablet aa4c873dd14d4c7e8fad1310a134502a (DEFAULT_TABLE table=test-table [id=35973c34db2c4e83bedbab9a0e15fcec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:42:57.228724 19856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa4c873dd14d4c7e8fad1310a134502a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:42:57.228338 19722 tablet_service.cc:1467] Processing CreateTablet for tablet aa4c873dd14d4c7e8fad1310a134502a (DEFAULT_TABLE table=test-table [id=35973c34db2c4e83bedbab9a0e15fcec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:42:57.228338 19990 tablet_service.cc:1467] Processing CreateTablet for tablet aa4c873dd14d4c7e8fad1310a134502a (DEFAULT_TABLE table=test-table [id=35973c34db2c4e83bedbab9a0e15fcec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:42:57.229389 19722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa4c873dd14d4c7e8fad1310a134502a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:42:57.229389 19990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa4c873dd14d4c7e8fad1310a134502a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:42:57.235391 20073 tablet_bootstrap.cc:492] T aa4c873dd14d4c7e8fad1310a134502a P e47fa4603d63429c9da8792553b713e0: Bootstrap starting.
I20260925 04:42:57.237001 20074 tablet_bootstrap.cc:492] T aa4c873dd14d4c7e8fad1310a134502a P 0cd6c4a2df1441d198a5e14539090b8b: Bootstrap starting.
I20260925 04:42:57.237416 20075 tablet_bootstrap.cc:492] T aa4c873dd14d4c7e8fad1310a134502a P 22191865f6ca4fb191c9f9ff5f64547d: Bootstrap starting.
I20260925 04:42:57.237881 20073 tablet_bootstrap.cc:654] T aa4c873dd14d4c7e8fad1310a134502a P e47fa4603d63429c9da8792553b713e0: Neither blocks nor log segments found. Creating new log.
I20260925 04:42:57.239164 20073 log.cc:824] T aa4c873dd14d4c7e8fad1310a134502a P e47fa4603d63429c9da8792553b713e0: Log is configured to *not* fsync() on all Append() calls
I20260925 04:42:57.239686 20075 tablet_bootstrap.cc:654] T aa4c873dd14d4c7e8fad1310a134502a P 22191865f6ca4fb191c9f9ff5f64547d: Neither blocks nor log segments found. Creating new log.
I20260925 04:42:57.241271 20073 tablet_bootstrap.cc:492] T aa4c873dd14d4c7e8fad1310a134502a P e47fa4603d63429c9da8792553b713e0: No bootstrap required, opened a new log
I20260925 04:42:57.241464 20073 ts_tablet_manager.cc:1397] T aa4c873dd14d4c7e8fad1310a134502a P e47fa4603d63429c9da8792553b713e0: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 04:42:57.242096 20075 log.cc:824] T aa4c873dd14d4c7e8fad1310a134502a P 22191865f6ca4fb191c9f9ff5f64547d: Log is configured to *not* fsync() on all Append() calls
I20260925 04:42:57.242162 20074 tablet_bootstrap.cc:654] T aa4c873dd14d4c7e8fad1310a134502a P 0cd6c4a2df1441d198a5e14539090b8b: Neither blocks nor log segments found. Creating new log.
I20260925 04:42:57.243348 20074 log.cc:824] T aa4c873dd14d4c7e8fad1310a134502a P 0cd6c4a2df1441d198a5e14539090b8b: Log is configured to *not* fsync() on all Append() calls
I20260925 04:42:57.244241 20075 tablet_bootstrap.cc:492] T aa4c873dd14d4c7e8fad1310a134502a P 22191865f6ca4fb191c9f9ff5f64547d: No bootstrap required, opened a new log
I20260925 04:42:57.244448 20075 ts_tablet_manager.cc:1397] T aa4c873dd14d4c7e8fad1310a134502a P 22191865f6ca4fb191c9f9ff5f64547d: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260925 04:42:57.244848 20073 raft_consensus.cc:348] T aa4c873dd14d4c7e8fad1310a134502a P e47fa4603d63429c9da8792553b713e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22191865f6ca4fb191c9f9ff5f64547d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44595 } } peers { permanent_uuid: "0cd6c4a2df1441d198a5e14539090b8b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40999 } } peers { permanent_uuid: "e47fa4603d63429c9da8792553b713e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44531 } }
I20260925 04:42:57.245090 20073 raft_consensus.cc:374] T aa4c873dd14d4c7e8fad1310a134502a P e47fa4603d63429c9da8792553b713e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:42:57.245159 20073 raft_consensus.cc:729] T aa4c873dd14d4c7e8fad1310a134502a P e47fa4603d63429c9da8792553b713e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47fa4603d63429c9da8792553b713e0, State: Initialized, Role: FOLLOWER
I20260925 04:42:57.245390 20074 tablet_bootstrap.cc:492] T aa4c873dd14d4c7e8fad1310a134502a P 0cd6c4a2df1441d198a5e14539090b8b: No bootstrap required, opened a new log
I20260925 04:42:57.245586 20074 ts_tablet_manager.cc:1397] T aa4c873dd14d4c7e8fad1310a134502a P 0cd6c4a2df1441d198a5e14539090b8b: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.002s
I20260925 04:42:57.245613 20073 consensus_queue.cc:260] T aa4c873dd14d4c7e8fad1310a134502a P e47fa4603d63429c9da8792553b713e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22191865f6ca4fb191c9f9ff5f64547d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44595 } } peers { permanent_uuid: "0cd6c4a2df1441d198a5e14539090b8b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40999 } } peers { permanent_uuid: "e47fa4603d63429c9da8792553b713e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44531 } }
I20260925 04:42:57.246299 19922 heartbeater.cc:503] Master 127.0.98.62:42211 was elected leader, sending a full tablet report...
I20260925 04:42:57.246569 20073 ts_tablet_manager.cc:1428] T aa4c873dd14d4c7e8fad1310a134502a P e47fa4603d63429c9da8792553b713e0: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 04:42:57.249419 20074 raft_consensus.cc:348] T aa4c873dd14d4c7e8fad1310a134502a P 0cd6c4a2df1441d198a5e14539090b8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22191865f6ca4fb191c9f9ff5f64547d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44595 } } peers { permanent_uuid: "0cd6c4a2df1441d198a5e14539090b8b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40999 } } peers { permanent_uuid: "e47fa4603d63429c9da8792553b713e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44531 } }
I20260925 04:42:57.249719 20074 raft_consensus.cc:374] T aa4c873dd14d4c7e8fad1310a134502a P 0cd6c4a2df1441d198a5e14539090b8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:42:57.249782 20074 raft_consensus.cc:729] T aa4c873dd14d4c7e8fad1310a134502a P 0cd6c4a2df1441d198a5e14539090b8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cd6c4a2df1441d198a5e14539090b8b, State: Initialized, Role: FOLLOWER
I20260925 04:42:57.250133 20075 raft_consensus.cc:348] T aa4c873dd14d4c7e8fad1310a134502a P 22191865f6ca4fb191c9f9ff5f64547d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22191865f6ca4fb191c9f9ff5f64547d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44595 } } peers { permanent_uuid: "0cd6c4a2df1441d198a5e14539090b8b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40999 } } peers { permanent_uuid: "e47fa4603d63429c9da8792553b713e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44531 } }
I20260925 04:42:57.250257 20074 consensus_queue.cc:260] T aa4c873dd14d4c7e8fad1310a134502a P 0cd6c4a2df1441d198a5e14539090b8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22191865f6ca4fb191c9f9ff5f64547d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44595 } } peers { permanent_uuid: "0cd6c4a2df1441d198a5e14539090b8b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40999 } } peers { permanent_uuid: "e47fa4603d63429c9da8792553b713e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44531 } }
I20260925 04:42:57.250388 20075 raft_consensus.cc:374] T aa4c873dd14d4c7e8fad1310a134502a P 22191865f6ca4fb191c9f9ff5f64547d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:42:57.250465 20075 raft_consensus.cc:729] T aa4c873dd14d4c7e8fad1310a134502a P 22191865f6ca4fb191c9f9ff5f64547d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22191865f6ca4fb191c9f9ff5f64547d, State: Initialized, Role: FOLLOWER
I20260925 04:42:57.251040 20075 consensus_queue.cc:260] T aa4c873dd14d4c7e8fad1310a134502a P 22191865f6ca4fb191c9f9ff5f64547d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22191865f6ca4fb191c9f9ff5f64547d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44595 } } peers { permanent_uuid: "0cd6c4a2df1441d198a5e14539090b8b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40999 } } peers { permanent_uuid: "e47fa4603d63429c9da8792553b713e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44531 } }
I20260925 04:42:57.251320 20074 ts_tablet_manager.cc:1428] T aa4c873dd14d4c7e8fad1310a134502a P 0cd6c4a2df1441d198a5e14539090b8b: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260925 04:42:57.251703 19788 heartbeater.cc:503] Master 127.0.98.62:42211 was elected leader, sending a full tablet report...
I20260925 04:42:57.251703 20056 heartbeater.cc:503] Master 127.0.98.62:42211 was elected leader, sending a full tablet report...
I20260925 04:42:57.251989 20075 ts_tablet_manager.cc:1428] T aa4c873dd14d4c7e8fad1310a134502a P 22191865f6ca4fb191c9f9ff5f64547d: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20260925 04:42:57.424256 20079 raft_consensus.cc:482] T aa4c873dd14d4c7e8fad1310a134502a P e47fa4603d63429c9da8792553b713e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:42:57.424500 20079 raft_consensus.cc:504] T aa4c873dd14d4c7e8fad1310a134502a P e47fa4603d63429c9da8792553b713e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22191865f6ca4fb191c9f9ff5f64547d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44595 } } peers { permanent_uuid: "0cd6c4a2df1441d198a5e14539090b8b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40999 } } peers { permanent_uuid: "e47fa4603d63429c9da8792553b713e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44531 } }
W20260925 04:42:57.425364 20057 tablet.cc:2369] T aa4c873dd14d4c7e8fad1310a134502a P 0cd6c4a2df1441d198a5e14539090b8b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:42:57.425806 20079 leader_election.cc:288] T aa4c873dd14d4c7e8fad1310a134502a P e47fa4603d63429c9da8792553b713e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22191865f6ca4fb191c9f9ff5f64547d (127.0.98.1:44595), 0cd6c4a2df1441d198a5e14539090b8b (127.0.98.3:40999)
I20260925 04:42:57.429625 20069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:57.426131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40637 (local address 127.0.98.1:44595)
0925 04:42:57.426260 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:57.426265 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:57.426607 (+ 342us) server_negotiation.cc:407] Connection header received
0925 04:42:57.426700 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:57.426703 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:57.426752 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:57.426833 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:42:57.427072 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:57.427592 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:57.428690 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:57.429066 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:57.429153 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:57.429216 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:57.429432 (+ 216us) server_negotiation.cc:299] Negotiation successful
0925 04:42:57.429499 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 04:42:57.429636 20072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:57.426134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40193 (local address 127.0.98.3:40999)
0925 04:42:57.426265 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:57.426268 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:57.426308 (+ 40us) server_negotiation.cc:407] Connection header received
0925 04:42:57.426475 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:57.426478 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:57.426572 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:57.426650 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:42:57.426961 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:57.427505 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:57.428668 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:57.429069 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:57.429161 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:57.429222 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:57.429448 (+ 226us) server_negotiation.cc:299] Negotiation successful
0925 04:42:57.429516 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 04:42:57.429653 20082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:57.426041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44595 (local address 127.0.98.2:40637)
0925 04:42:57.426483 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:57.426496 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:42:57.426613 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:42:57.426848 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:42:57.426850 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:42:57.426864 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:42:57.426969 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:57.426975 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:57.427718 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:57.427721 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:57.428574 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:57.428582 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:57.429183 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:57.429187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:57.429315 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:57.429341 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 04:42:57.429400 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 04:42:57.429473 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"mutex_wait_us":217,"thread_start_us":91,"threads_started":1}
I20260925 04:42:57.429641 19924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:57.426041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40999 (local address 127.0.98.2:40193)
0925 04:42:57.426238 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:57.426253 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:42:57.426357 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:42:57.426692 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:42:57.426695 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:42:57.426715 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:42:57.426852 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:57.426859 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:57.427637 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:57.427640 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:57.428547 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:57.428555 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:57.429184 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:57.429187 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:57.429338 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:57.429355 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 04:42:57.429423 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 04:42:57.429493 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":87}
I20260925 04:42:57.430404 19742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa4c873dd14d4c7e8fad1310a134502a" candidate_uuid: "e47fa4603d63429c9da8792553b713e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22191865f6ca4fb191c9f9ff5f64547d" is_pre_election: true
I20260925 04:42:57.430405 20010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa4c873dd14d4c7e8fad1310a134502a" candidate_uuid: "e47fa4603d63429c9da8792553b713e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cd6c4a2df1441d198a5e14539090b8b" is_pre_election: true
I20260925 04:42:57.430728 19742 raft_consensus.cc:2454] T aa4c873dd14d4c7e8fad1310a134502a P 22191865f6ca4fb191c9f9ff5f64547d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e47fa4603d63429c9da8792553b713e0 in term 0.
I20260925 04:42:57.430738 20010 raft_consensus.cc:2454] T aa4c873dd14d4c7e8fad1310a134502a P 0cd6c4a2df1441d198a5e14539090b8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e47fa4603d63429c9da8792553b713e0 in term 0.
I20260925 04:42:57.431252 19812 leader_election.cc:302] T aa4c873dd14d4c7e8fad1310a134502a P e47fa4603d63429c9da8792553b713e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22191865f6ca4fb191c9f9ff5f64547d, e47fa4603d63429c9da8792553b713e0; no voters:
I20260925 04:42:57.431526 20079 raft_consensus.cc:2781] T aa4c873dd14d4c7e8fad1310a134502a P e47fa4603d63429c9da8792553b713e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:42:57.431631 20079 raft_consensus.cc:482] T aa4c873dd14d4c7e8fad1310a134502a P e47fa4603d63429c9da8792553b713e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:42:57.431684 20079 raft_consensus.cc:3037] T aa4c873dd14d4c7e8fad1310a134502a P e47fa4603d63429c9da8792553b713e0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:42:57.432693 20079 raft_consensus.cc:504] T aa4c873dd14d4c7e8fad1310a134502a P e47fa4603d63429c9da8792553b713e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22191865f6ca4fb191c9f9ff5f64547d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44595 } } peers { permanent_uuid: "0cd6c4a2df1441d198a5e14539090b8b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40999 } } peers { permanent_uuid: "e47fa4603d63429c9da8792553b713e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44531 } }
I20260925 04:42:57.433048 20079 leader_election.cc:288] T aa4c873dd14d4c7e8fad1310a134502a P e47fa4603d63429c9da8792553b713e0 [CANDIDATE]: Term 1 election: Requested vote from peers 22191865f6ca4fb191c9f9ff5f64547d (127.0.98.1:44595), 0cd6c4a2df1441d198a5e14539090b8b (127.0.98.3:40999)
I20260925 04:42:57.433442 19742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa4c873dd14d4c7e8fad1310a134502a" candidate_uuid: "e47fa4603d63429c9da8792553b713e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22191865f6ca4fb191c9f9ff5f64547d"
I20260925 04:42:57.433472 20010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa4c873dd14d4c7e8fad1310a134502a" candidate_uuid: "e47fa4603d63429c9da8792553b713e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cd6c4a2df1441d198a5e14539090b8b"
I20260925 04:42:57.433568 19742 raft_consensus.cc:3037] T aa4c873dd14d4c7e8fad1310a134502a P 22191865f6ca4fb191c9f9ff5f64547d [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:42:57.433578 20010 raft_consensus.cc:3037] T aa4c873dd14d4c7e8fad1310a134502a P 0cd6c4a2df1441d198a5e14539090b8b [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:42:57.434580 20010 raft_consensus.cc:2454] T aa4c873dd14d4c7e8fad1310a134502a P 0cd6c4a2df1441d198a5e14539090b8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e47fa4603d63429c9da8792553b713e0 in term 1.
I20260925 04:42:57.434629 19742 raft_consensus.cc:2454] T aa4c873dd14d4c7e8fad1310a134502a P 22191865f6ca4fb191c9f9ff5f64547d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e47fa4603d63429c9da8792553b713e0 in term 1.
I20260925 04:42:57.434912 19810 leader_election.cc:302] T aa4c873dd14d4c7e8fad1310a134502a P e47fa4603d63429c9da8792553b713e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cd6c4a2df1441d198a5e14539090b8b, e47fa4603d63429c9da8792553b713e0; no voters:
I20260925 04:42:57.435086 20079 raft_consensus.cc:2781] T aa4c873dd14d4c7e8fad1310a134502a P e47fa4603d63429c9da8792553b713e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:42:57.435307 20079 raft_consensus.cc:686] T aa4c873dd14d4c7e8fad1310a134502a P e47fa4603d63429c9da8792553b713e0 [term 1 LEADER]: Becoming Leader. State: Replica: e47fa4603d63429c9da8792553b713e0, State: Running, Role: LEADER
I20260925 04:42:57.435689 20079 consensus_queue.cc:237] T aa4c873dd14d4c7e8fad1310a134502a P e47fa4603d63429c9da8792553b713e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22191865f6ca4fb191c9f9ff5f64547d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44595 } } peers { permanent_uuid: "0cd6c4a2df1441d198a5e14539090b8b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40999 } } peers { permanent_uuid: "e47fa4603d63429c9da8792553b713e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44531 } }
I20260925 04:42:57.438949 19596 catalog_manager.cc:5495] T aa4c873dd14d4c7e8fad1310a134502a P e47fa4603d63429c9da8792553b713e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to e47fa4603d63429c9da8792553b713e0 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "e47fa4603d63429c9da8792553b713e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22191865f6ca4fb191c9f9ff5f64547d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cd6c4a2df1441d198a5e14539090b8b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e47fa4603d63429c9da8792553b713e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44531 } health_report { overall_health: HEALTHY } } }
W20260925 04:42:57.453104 19923 tablet.cc:2369] T aa4c873dd14d4c7e8fad1310a134502a P e47fa4603d63429c9da8792553b713e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:42:57.461881 20071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:57.458281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55650 (local address 127.0.98.2:44531)
0925 04:42:57.458414 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:57.458418 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:57.458432 (+ 14us) server_negotiation.cc:407] Connection header received
0925 04:42:57.458553 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:57.458555 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:57.458609 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:57.458693 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:42:57.458917 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:57.459432 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:57.460337 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:57.460572 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:57.460665 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:57.461017 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:42:57.461460 (+ 443us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:42:57.461570 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:57.461652 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 04:42:57.461716 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 04:42:57.467779 20010 raft_consensus.cc:1261] T aa4c873dd14d4c7e8fad1310a134502a P 0cd6c4a2df1441d198a5e14539090b8b [term 1 FOLLOWER]: Refusing update from remote peer e47fa4603d63429c9da8792553b713e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:42:57.467779 19742 raft_consensus.cc:1261] T aa4c873dd14d4c7e8fad1310a134502a P 22191865f6ca4fb191c9f9ff5f64547d [term 1 FOLLOWER]: Refusing update from remote peer e47fa4603d63429c9da8792553b713e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:42:57.468750 20083 consensus_queue.cc:1035] T aa4c873dd14d4c7e8fad1310a134502a P e47fa4603d63429c9da8792553b713e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22191865f6ca4fb191c9f9ff5f64547d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:42:57.468955 20079 consensus_queue.cc:1035] T aa4c873dd14d4c7e8fad1310a134502a P e47fa4603d63429c9da8792553b713e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cd6c4a2df1441d198a5e14539090b8b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:42:57.477445 20086 mvcc.cc:205] Tried to move back new op lower bound from 7333115402100408320 to 7333115401979445248. Current Snapshot: MvccSnapshot[applied={T|T < 7333115402100408320}]
I20260925 04:42:57.479581 20087 mvcc.cc:205] Tried to move back new op lower bound from 7333115402100408320 to 7333115401979445248. Current Snapshot: MvccSnapshot[applied={T|T < 7333115402100408320}]
I20260925 04:42:57.482540 20088 mvcc.cc:205] Tried to move back new op lower bound from 7333115402100408320 to 7333115401979445248. Current Snapshot: MvccSnapshot[applied={T|T < 7333115402100408320 or (T in {7333115402100408320})}]
I20260925 04:42:57.486876 19596 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:42:57.487054 19596 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:42:57.489504 19596 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 57f55d294c1741ec8af839df0a26c033: Sending DeleteTablet for 3 replicas of tablet aa4c873dd14d4c7e8fad1310a134502a
I20260925 04:42:57.490500 19856 tablet_service.cc:1514] Processing DeleteTablet for tablet aa4c873dd14d4c7e8fad1310a134502a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:42:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55634
I20260925 04:42:57.490494 19990 tablet_service.cc:1514] Processing DeleteTablet for tablet aa4c873dd14d4c7e8fad1310a134502a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:42:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58904
I20260925 04:42:57.490536 19722 tablet_service.cc:1514] Processing DeleteTablet for tablet aa4c873dd14d4c7e8fad1310a134502a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:42:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50040
I20260925 04:42:57.491143 20097 tablet_replica.cc:330] T aa4c873dd14d4c7e8fad1310a134502a P 0cd6c4a2df1441d198a5e14539090b8b: stopping tablet replica
I20260925 04:42:57.491513 20097 raft_consensus.cc:2229] T aa4c873dd14d4c7e8fad1310a134502a P 0cd6c4a2df1441d198a5e14539090b8b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:42:57.491679 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 19659
I20260925 04:42:57.491834 20096 tablet_replica.cc:330] T aa4c873dd14d4c7e8fad1310a134502a P e47fa4603d63429c9da8792553b713e0: stopping tablet replica
I20260925 04:42:57.492110 20096 raft_consensus.cc:2229] T aa4c873dd14d4c7e8fad1310a134502a P e47fa4603d63429c9da8792553b713e0 [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:42:57.492528 20096 raft_consensus.cc:2258] T aa4c873dd14d4c7e8fad1310a134502a P e47fa4603d63429c9da8792553b713e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:42:57.493022 20097 raft_consensus.cc:2258] T aa4c873dd14d4c7e8fad1310a134502a P 0cd6c4a2df1441d198a5e14539090b8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:42:57.494616 20096 ts_tablet_manager.cc:1905] T aa4c873dd14d4c7e8fad1310a134502a P e47fa4603d63429c9da8792553b713e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:42:57.497184 20097 ts_tablet_manager.cc:1905] T aa4c873dd14d4c7e8fad1310a134502a P 0cd6c4a2df1441d198a5e14539090b8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:42:57.497459 20096 ts_tablet_manager.cc:1918] T aa4c873dd14d4c7e8fad1310a134502a P e47fa4603d63429c9da8792553b713e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:42:57.497546 20096 log.cc:1196] T aa4c873dd14d4c7e8fad1310a134502a P e47fa4603d63429c9da8792553b713e0: Deleting WAL directory at /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/minicluster-data/ts-1/wal/wals/aa4c873dd14d4c7e8fad1310a134502a
I20260925 04:42:57.497833 20096 ts_tablet_manager.cc:1939] T aa4c873dd14d4c7e8fad1310a134502a P e47fa4603d63429c9da8792553b713e0: Deleting consensus metadata
I20260925 04:42:57.499128 19580 catalog_manager.cc:4841] TS e47fa4603d63429c9da8792553b713e0 (127.0.98.2:44531): tablet aa4c873dd14d4c7e8fad1310a134502a (table test-table [id=35973c34db2c4e83bedbab9a0e15fcec]) successfully deleted
I20260925 04:42:57.500218 20097 ts_tablet_manager.cc:1918] T aa4c873dd14d4c7e8fad1310a134502a P 0cd6c4a2df1441d198a5e14539090b8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:42:57.500324 20097 log.cc:1196] T aa4c873dd14d4c7e8fad1310a134502a P 0cd6c4a2df1441d198a5e14539090b8b: Deleting WAL directory at /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311123478858-392-0/minicluster-data/ts-2/wal/wals/aa4c873dd14d4c7e8fad1310a134502a
I20260925 04:42:57.500635 20097 ts_tablet_manager.cc:1939] T aa4c873dd14d4c7e8fad1310a134502a P 0cd6c4a2df1441d198a5e14539090b8b: Deleting consensus metadata
W20260925 04:42:57.500896 19583 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44595 (error 108)
I20260925 04:42:57.501852 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 19793
I20260925 04:42:57.502158 20067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:57.501719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44595 (local address 127.0.0.1:50050)
0925 04:42:57.501943 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:57.502075 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 04:42:57.502441 19581 catalog_manager.cc:4841] TS 0cd6c4a2df1441d198a5e14539090b8b (127.0.98.3:40999): tablet aa4c873dd14d4c7e8fad1310a134502a (table test-table [id=35973c34db2c4e83bedbab9a0e15fcec]) successfully deleted
W20260925 04:42:57.502681 19583 catalog_manager.cc:4584] TS 22191865f6ca4fb191c9f9ff5f64547d (127.0.98.1:44595): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet aa4c873dd14d4c7e8fad1310a134502a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44595: connect: Connection refused (error 111)
I20260925 04:42:57.507174 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 19927
I20260925 04:42:57.512063 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 19564
2026-09-25T04:42:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2320 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-25T04:42:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:42:57Z Disabled control of system clock
I20260925 04:42:57.539891 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311123478858-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311123478858-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311123478858-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35925
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35925
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 04:42:57.651578 20102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:42:57.651895 20102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:42:57.651932 20102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:42:57.656042 20102 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:42:57.656113 20102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:42:57.656127 20102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:42:57.656143 20102 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:42:57.656158 20102 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:42:57.661080 20102 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311123478858-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311123478858-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35925
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311123478858-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:42:57.662170 20102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:42:57.663528 20102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:42:57.672317 20107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:42:57.672377 20110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:42:57.672317 20108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:42:57.672785 20102 server_base.cc:1034] running on GCE node
I20260925 04:42:57.673342 20102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:42:57.674521 20102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:42:57.675726 20102 hybrid_clock.cc:648] HybridClock initialized: now 1790311377675699 us; error 45 us; skew 500 ppm
I20260925 04:42:57.677690 20102 webserver.cc:460] Webserver started at http://127.0.98.62:44975/ using document root <none> and password file <none>
I20260925 04:42:57.678257 20102 fs_manager.cc:359] Metadata directory not provided
I20260925 04:42:57.678319 20102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:42:57.678488 20102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:42:57.680729 20102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311123478858-392-0/minicluster-data/master-0/data/instance:
uuid: "05f61fab032f495c8faa7dd631f17594"
format_stamp: "Formatted at 2026-09-25 04:42:57 on dist-test-slave-dc95"
server_key: "9cf6ec4c547e43826ec57105015aecd7"
server_key_iv: "73572f3f2f9fb0805cdd23d3ab8a2308"
server_key_version: "encryptionkey@0"
I20260925 04:42:57.681277 20102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311123478858-392-0/minicluster-data/master-0/wal/instance:
uuid: "05f61fab032f495c8faa7dd631f17594"
format_stamp: "Formatted at 2026-09-25 04:42:57 on dist-test-slave-dc95"
server_key: "9cf6ec4c547e43826ec57105015aecd7"
server_key_iv: "73572f3f2f9fb0805cdd23d3ab8a2308"
server_key_version: "encryptionkey@0"
I20260925 04:42:57.685314 20102 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:42:57.688179 20116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:57.689625 20102 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:42:57.689777 20102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311123478858-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311123478858-392-0/minicluster-data/master-0/wal
uuid: "05f61fab032f495c8faa7dd631f17594"
format_stamp: "Formatted at 2026-09-25 04:42:57 on dist-test-slave-dc95"
server_key: "9cf6ec4c547e43826ec57105015aecd7"
server_key_iv: "73572f3f2f9fb0805cdd23d3ab8a2308"
server_key_version: "encryptionkey@0"
I20260925 04:42:57.689899 20102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311123478858-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311123478858-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311123478858-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:42:57.709005 20102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:42:57.709744 20102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:42:57.709923 20102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:42:57.716899 20102 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35925
I20260925 04:42:57.716914 20168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35925 every 8 connection(s)
I20260925 04:42:57.717921 20102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
I20260925 04:42:57.721131 20169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:42:57.725927 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 20102
I20260925 04:42:57.726058 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311123478858-392-0/minicluster-data/master-0/wal/instance
I20260925 04:42:57.726310 392 external_mini_cluster.cc:1428] Setting key b6dcc6667e5469a844ef5b2f2b70c6fd
I20260925 04:42:57.726962 20169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05f61fab032f495c8faa7dd631f17594: Bootstrap starting.
I20260925 04:42:57.729449 20169 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05f61fab032f495c8faa7dd631f17594: Neither blocks nor log segments found. Creating new log.
I20260925 04:42:57.730196 20169 log.cc:824] T 00000000000000000000000000000000 P 05f61fab032f495c8faa7dd631f17594: Log is configured to *not* fsync() on all Append() calls
I20260925 04:42:57.732323 20169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05f61fab032f495c8faa7dd631f17594: No bootstrap required, opened a new log
I20260925 04:42:57.735874 20172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:57.727743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58350 (local address 127.0.98.62:35925)
0925 04:42:57.728448 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:57.728467 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:57.728505 (+ 38us) server_negotiation.cc:407] Connection header received
0925 04:42:57.729171 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:57.729189 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:57.729637 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:57.730026 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:42:57.730484 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:57.731746 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:57.732657 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:57.733157 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:57.733311 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:57.734080 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:42:57.734085 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:42:57.734099 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:42:57.734490 (+ 391us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:42:57.734545 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:42:57.734566 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:42:57.734653 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:57.734786 (+ 133us) server_negotiation.cc:299] Negotiation successful
0925 04:42:57.735056 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":135,"threads_started":1}
I20260925 04:42:57.735934 20169 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05f61fab032f495c8faa7dd631f17594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f61fab032f495c8faa7dd631f17594" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35925 } }
I20260925 04:42:57.736212 20169 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05f61fab032f495c8faa7dd631f17594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:42:57.736268 20169 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05f61fab032f495c8faa7dd631f17594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f61fab032f495c8faa7dd631f17594, State: Initialized, Role: FOLLOWER
I20260925 04:42:57.736708 20169 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05f61fab032f495c8faa7dd631f17594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f61fab032f495c8faa7dd631f17594" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35925 } }
I20260925 04:42:57.736840 20169 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05f61fab032f495c8faa7dd631f17594 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:42:57.736898 20169 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05f61fab032f495c8faa7dd631f17594 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:42:57.736994 20169 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05f61fab032f495c8faa7dd631f17594 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:42:57.737893 20169 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05f61fab032f495c8faa7dd631f17594 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f61fab032f495c8faa7dd631f17594" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35925 } }
I20260925 04:42:57.738245 20169 leader_election.cc:302] T 00000000000000000000000000000000 P 05f61fab032f495c8faa7dd631f17594 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05f61fab032f495c8faa7dd631f17594; no voters:
I20260925 04:42:57.738510 20169 leader_election.cc:288] T 00000000000000000000000000000000 P 05f61fab032f495c8faa7dd631f17594 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:42:57.738659 20174 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05f61fab032f495c8faa7dd631f17594 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:42:57.738859 20174 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05f61fab032f495c8faa7dd631f17594 [term 1 LEADER]: Becoming Leader. State: Replica: 05f61fab032f495c8faa7dd631f17594, State: Running, Role: LEADER
I20260925 04:42:57.739172 20174 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05f61fab032f495c8faa7dd631f17594 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f61fab032f495c8faa7dd631f17594" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35925 } }
I20260925 04:42:57.739686 20169 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05f61fab032f495c8faa7dd631f17594 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:42:57.740872 20176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05f61fab032f495c8faa7dd631f17594 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05f61fab032f495c8faa7dd631f17594. Latest consensus state: current_term: 1 leader_uuid: "05f61fab032f495c8faa7dd631f17594" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f61fab032f495c8faa7dd631f17594" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35925 } } }
I20260925 04:42:57.741029 20176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05f61fab032f495c8faa7dd631f17594 [sys.catalog]: This master's current role is: LEADER
I20260925 04:42:57.740869 20175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05f61fab032f495c8faa7dd631f17594 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05f61fab032f495c8faa7dd631f17594" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f61fab032f495c8faa7dd631f17594" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35925 } } }
I20260925 04:42:57.741185 20175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05f61fab032f495c8faa7dd631f17594 [sys.catalog]: This master's current role is: LEADER
W20260925 04:42:57.744441 20189 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 05f61fab032f495c8faa7dd631f17594: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 04:42:57.744513 20189 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 04:42:57.744582 20184 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:42:57.745280 20184 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:42:57.750976 20184 catalog_manager.cc:1347] Generated new cluster ID: 6074632244454dc6896b9bcdfca6dd86
I20260925 04:42:57.751065 20184 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:42:57.764453 20184 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:42:57.765429 20184 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:42:57.774593 20184 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05f61fab032f495c8faa7dd631f17594: Generated new TSK 0
I20260925 04:42:57.775419 20184 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:42:57.778506 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311123478858-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311123478858-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311123478858-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35925
--builtin_ntp_servers=127.0.98.20:35889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 04:42:57.895323 20193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:42:57.895648 20193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:42:57.895699 20193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:42:57.899937 20193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:42:57.900002 20193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:42:57.900059 20193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 04:42:57.905048 20193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311123478858-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311123478858-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311123478858-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:42:57.906080 20193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:42:57.907387 20193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:42:57.916919 20199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:42:57.917065 20193 server_base.cc:1034] running on GCE node
W20260925 04:42:57.916880 20201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:42:57.916893 20198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:42:57.917608 20193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:42:57.918232 20193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:42:57.919400 20193 hybrid_clock.cc:648] HybridClock initialized: now 1790311377919370 us; error 42 us; skew 500 ppm
I20260925 04:42:57.921219 20193 webserver.cc:460] Webserver started at http://127.0.98.1:42153/ using document root <none> and password file <none>
I20260925 04:42:57.921763 20193 fs_manager.cc:359] Metadata directory not provided
I20260925 04:42:57.921823 20193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:42:57.922039 20193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:42:57.924183 20193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311123478858-392-0/minicluster-data/ts-0/data/instance:
uuid: "3b6cbdad86bc46a1a691ba97b7fb18f8"
format_stamp: "Formatted at 2026-09-25 04:42:57 on dist-test-slave-dc95"
server_key: "b5936a3651317c13542ee36df2d6c9f0"
server_key_iv: "faadfdc8088fe7fac623d2339d71c574"
server_key_version: "encryptionkey@0"
I20260925 04:42:57.924723 20193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311123478858-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3b6cbdad86bc46a1a691ba97b7fb18f8"
format_stamp: "Formatted at 2026-09-25 04:42:57 on dist-test-slave-dc95"
server_key: "b5936a3651317c13542ee36df2d6c9f0"
server_key_iv: "faadfdc8088fe7fac623d2339d71c574"
server_key_version: "encryptionkey@0"
I20260925 04:42:57.928418 20193 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:42:57.930853 20207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:57.932287 20193 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:42:57.932433 20193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311123478858-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311123478858-392-0/minicluster-data/ts-0/wal
uuid: "3b6cbdad86bc46a1a691ba97b7fb18f8"
format_stamp: "Formatted at 2026-09-25 04:42:57 on dist-test-slave-dc95"
server_key: "b5936a3651317c13542ee36df2d6c9f0"
server_key_iv: "faadfdc8088fe7fac623d2339d71c574"
server_key_version: "encryptionkey@0"
I20260925 04:42:57.932556 20193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311123478858-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311123478858-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311123478858-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:42:57.946426 20193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:42:57.947153 20193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:42:57.947340 20193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:42:57.947988 20193 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:42:57.948948 20193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:42:57.949004 20193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:57.949046 20193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:42:57.949074 20193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:57.957731 20193 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42551
I20260925 04:42:57.957756 20320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42551 every 8 connection(s)
I20260925 04:42:57.958678 20193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311123478858-392-0/minicluster-data/ts-0/data/info.pb
I20260925 04:42:57.964830 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 20193
I20260925 04:42:57.964963 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311123478858-392-0/minicluster-data/ts-0/wal/instance
I20260925 04:42:57.965293 392 external_mini_cluster.cc:1428] Setting key 9fb9401c7b1b56397e04c947d8fce3da
I20260925 04:42:57.967672 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311123478858-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311123478858-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311123478858-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35925
--builtin_ntp_servers=127.0.98.20:35889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 04:42:57.969190 20172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:57.960550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43563 (local address 127.0.98.62:35925)
0925 04:42:57.960697 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:57.960701 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:57.961949 (+ 1248us) server_negotiation.cc:407] Connection header received
0925 04:42:57.963373 (+ 1424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:57.963376 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:57.963426 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:57.963515 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:42:57.964600 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:57.965381 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:57.966998 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:57.967309 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:57.967444 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:57.968653 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:42:57.968657 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:42:57.968660 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:42:57.968674 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:42:57.968697 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:42:57.968708 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:42:57.968781 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:57.969008 (+ 227us) server_negotiation.cc:299] Negotiation successful
0925 04:42:57.969070 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 04:42:57.970352 20323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:57.960955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35925 (local address 127.0.98.1:43563)
0925 04:42:57.961737 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:57.961806 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0925 04:42:57.963151 (+ 1345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:42:57.963765 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:42:57.963773 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:42:57.963871 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:42:57.964440 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:57.964457 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:57.965544 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:57.965547 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:57.966839 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:57.966852 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:57.967494 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:57.967498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:57.967670 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:57.968352 (+ 682us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:42:57.968354 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:42:57.968400 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:42:57.968489 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:42:57.968838 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:42:57.968849 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:42:57.968856 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 04:42:57.969002 (+ 146us) client_negotiation.cc:240] Negotiation successful
0925 04:42:57.969334 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":115,"threads_started":1}
I20260925 04:42:57.971832 20321 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35925
I20260925 04:42:57.972080 20321 heartbeater.cc:464] Registering TS with master...
I20260925 04:42:57.972633 20321 heartbeater.cc:511] Master 127.0.98.62:35925 requested a full tablet report, sending...
I20260925 04:42:57.974186 20133 ts_manager.cc:194] Registered new tserver with Master: 3b6cbdad86bc46a1a691ba97b7fb18f8 (127.0.98.1:42551)
I20260925 04:42:57.975335 20133 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:43563
W20260925 04:42:58.079849 20324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:42:58.080150 20324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:42:58.080205 20324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:42:58.084371 20324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:42:58.084438 20324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:42:58.084517 20324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 04:42:58.089588 20324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311123478858-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311123478858-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311123478858-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:42:58.090778 20324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:42:58.092254 20324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:42:58.102276 20330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:42:58.102267 20329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:42:58.102463 20324 server_base.cc:1034] running on GCE node
W20260925 04:42:58.102279 20332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:42:58.102977 20324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:42:58.103726 20324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:42:58.104957 20324 hybrid_clock.cc:648] HybridClock initialized: now 1790311378104937 us; error 32 us; skew 500 ppm
I20260925 04:42:58.106793 20324 webserver.cc:460] Webserver started at http://127.0.98.2:42191/ using document root <none> and password file <none>
I20260925 04:42:58.107375 20324 fs_manager.cc:359] Metadata directory not provided
I20260925 04:42:58.107436 20324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:42:58.107688 20324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:42:58.109815 20324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311123478858-392-0/minicluster-data/ts-1/data/instance:
uuid: "fd00ae08be8a4b13bd7c60ea0d45cb49"
format_stamp: "Formatted at 2026-09-25 04:42:58 on dist-test-slave-dc95"
server_key: "a2d007e8819e5952fd42384d7bf8ed06"
server_key_iv: "c5afb51bee6d08739c648154400656ff"
server_key_version: "encryptionkey@0"
I20260925 04:42:58.110383 20324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311123478858-392-0/minicluster-data/ts-1/wal/instance:
uuid: "fd00ae08be8a4b13bd7c60ea0d45cb49"
format_stamp: "Formatted at 2026-09-25 04:42:58 on dist-test-slave-dc95"
server_key: "a2d007e8819e5952fd42384d7bf8ed06"
server_key_iv: "c5afb51bee6d08739c648154400656ff"
server_key_version: "encryptionkey@0"
I20260925 04:42:58.114288 20324 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 04:42:58.117030 20338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:58.118462 20324 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 04:42:58.118585 20324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311123478858-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311123478858-392-0/minicluster-data/ts-1/wal
uuid: "fd00ae08be8a4b13bd7c60ea0d45cb49"
format_stamp: "Formatted at 2026-09-25 04:42:58 on dist-test-slave-dc95"
server_key: "a2d007e8819e5952fd42384d7bf8ed06"
server_key_iv: "c5afb51bee6d08739c648154400656ff"
server_key_version: "encryptionkey@0"
I20260925 04:42:58.118728 20324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311123478858-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311123478858-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311123478858-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:42:58.137579 20324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:42:58.138304 20324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:42:58.138479 20324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:42:58.139081 20324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:42:58.140136 20324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:42:58.140225 20324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:58.140282 20324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:42:58.140313 20324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:58.148782 20324 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42505
I20260925 04:42:58.148844 20451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42505 every 8 connection(s)
I20260925 04:42:58.149688 20324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311123478858-392-0/minicluster-data/ts-1/data/info.pb
I20260925 04:42:58.154750 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 20324
I20260925 04:42:58.154887 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311123478858-392-0/minicluster-data/ts-1/wal/instance
I20260925 04:42:58.155148 392 external_mini_cluster.cc:1428] Setting key 88fa2dc2abb47378d768126751d2c72c
I20260925 04:42:58.157058 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311123478858-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311123478858-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311123478858-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35925
--builtin_ntp_servers=127.0.98.20:35889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 04:42:58.159782 20172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:58.151580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56535 (local address 127.0.98.62:35925)
0925 04:42:58.151760 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:58.151765 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:58.152917 (+ 1152us) server_negotiation.cc:407] Connection header received
0925 04:42:58.154282 (+ 1365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:58.154285 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:58.154340 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:58.154421 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:42:58.155522 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:58.156203 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:58.157514 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:58.157830 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:58.157935 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:58.159195 (+ 1260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:42:58.159199 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:42:58.159201 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:42:58.159215 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:42:58.159237 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:42:58.159245 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:42:58.159305 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:58.159495 (+ 190us) server_negotiation.cc:299] Negotiation successful
0925 04:42:58.159560 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 04:42:58.160897 20454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:58.152018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35925 (local address 127.0.98.2:56535)
0925 04:42:58.152719 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:58.152762 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 04:42:58.154046 (+ 1284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:42:58.154650 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:42:58.154658 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:42:58.154759 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:42:58.155357 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:58.155375 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:58.156334 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:58.156337 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:58.157386 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:58.157398 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:58.157987 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:58.157990 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:58.158147 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:58.158849 (+ 702us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:42:58.158851 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:42:58.158913 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:42:58.159007 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:42:58.159330 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:42:58.159339 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:42:58.159345 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 04:42:58.159479 (+ 134us) client_negotiation.cc:240] Negotiation successful
0925 04:42:58.159830 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":125,"threads_started":1}
I20260925 04:42:58.162287 20452 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35925
I20260925 04:42:58.162532 20452 heartbeater.cc:464] Registering TS with master...
I20260925 04:42:58.163082 20452 heartbeater.cc:511] Master 127.0.98.62:35925 requested a full tablet report, sending...
I20260925 04:42:58.164284 20133 ts_manager.cc:194] Registered new tserver with Master: fd00ae08be8a4b13bd7c60ea0d45cb49 (127.0.98.2:42505)
I20260925 04:42:58.164891 20133 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:56535
W20260925 04:42:58.272665 20455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:42:58.272949 20455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:42:58.272989 20455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:42:58.277163 20455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:42:58.277228 20455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:42:58.277288 20455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 04:42:58.282292 20455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311123478858-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311123478858-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311123478858-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:42:58.283464 20455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:42:58.284924 20455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:42:58.294921 20461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:42:58.294984 20460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:42:58.295007 20455 server_base.cc:1034] running on GCE node
W20260925 04:42:58.295249 20463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:42:58.295706 20455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:42:58.296379 20455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:42:58.297557 20455 hybrid_clock.cc:648] HybridClock initialized: now 1790311378297517 us; error 64 us; skew 500 ppm
I20260925 04:42:58.299767 20455 webserver.cc:460] Webserver started at http://127.0.98.3:46203/ using document root <none> and password file <none>
I20260925 04:42:58.300355 20455 fs_manager.cc:359] Metadata directory not provided
I20260925 04:42:58.300423 20455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:42:58.300616 20455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:42:58.302821 20455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311123478858-392-0/minicluster-data/ts-2/data/instance:
uuid: "9a38b2f488de4d7d8a6b26c34c737466"
format_stamp: "Formatted at 2026-09-25 04:42:58 on dist-test-slave-dc95"
server_key: "7bb69a60f49118cd0c2e923bb1ea3930"
server_key_iv: "3ac73c9256e95c909b8b9f122a5340ad"
server_key_version: "encryptionkey@0"
I20260925 04:42:58.303401 20455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311123478858-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9a38b2f488de4d7d8a6b26c34c737466"
format_stamp: "Formatted at 2026-09-25 04:42:58 on dist-test-slave-dc95"
server_key: "7bb69a60f49118cd0c2e923bb1ea3930"
server_key_iv: "3ac73c9256e95c909b8b9f122a5340ad"
server_key_version: "encryptionkey@0"
I20260925 04:42:58.307454 20455 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 04:42:58.310271 20469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:58.311735 20455 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:42:58.311868 20455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311123478858-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311123478858-392-0/minicluster-data/ts-2/wal
uuid: "9a38b2f488de4d7d8a6b26c34c737466"
format_stamp: "Formatted at 2026-09-25 04:42:58 on dist-test-slave-dc95"
server_key: "7bb69a60f49118cd0c2e923bb1ea3930"
server_key_iv: "3ac73c9256e95c909b8b9f122a5340ad"
server_key_version: "encryptionkey@0"
I20260925 04:42:58.311993 20455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311123478858-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311123478858-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311123478858-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:42:58.330838 20455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:42:58.331550 20455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:42:58.331780 20455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:42:58.332367 20455 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:42:58.333374 20455 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:42:58.333426 20455 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:58.333467 20455 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:42:58.333482 20455 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:42:58.341933 20455 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37179
I20260925 04:42:58.341956 20582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37179 every 8 connection(s)
I20260925 04:42:58.342876 20455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311123478858-392-0/minicluster-data/ts-2/data/info.pb
I20260925 04:42:58.344543 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 20455
I20260925 04:42:58.344645 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311123478858-392-0/minicluster-data/ts-2/wal/instance
I20260925 04:42:58.344959 392 external_mini_cluster.cc:1428] Setting key 519cb04adebb32e72604b8119bc0131a
I20260925 04:42:58.352360 20172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:58.344833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38579 (local address 127.0.98.62:35925)
0925 04:42:58.344982 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:58.344987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:58.346240 (+ 1253us) server_negotiation.cc:407] Connection header received
0925 04:42:58.347790 (+ 1550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:58.347793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:58.347842 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:58.347906 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:42:58.348848 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:58.349371 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:58.350547 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:58.350789 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:58.350867 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:58.351952 (+ 1085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:42:58.351955 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:42:58.351957 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:42:58.351970 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:42:58.351988 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:42:58.351996 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:42:58.352051 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:58.352161 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 04:42:58.352237 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 04:42:58.353289 20585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:58.345223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35925 (local address 127.0.98.3:38579)
0925 04:42:58.346069 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:58.346128 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 04:42:58.347552 (+ 1424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:42:58.348071 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:42:58.348078 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:42:58.348176 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:42:58.348670 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:58.348735 (+ 65us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:58.349490 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:58.349494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:58.350434 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:58.350444 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:58.350929 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:58.350932 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:58.351086 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:58.351692 (+ 606us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:42:58.351694 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:42:58.351737 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:42:58.351806 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:42:58.352058 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:42:58.352063 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:42:58.352067 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 04:42:58.352159 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 04:42:58.352396 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":137,"threads_started":1}
I20260925 04:42:58.354506 20583 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35925
I20260925 04:42:58.354754 20583 heartbeater.cc:464] Registering TS with master...
I20260925 04:42:58.355299 20583 heartbeater.cc:511] Master 127.0.98.62:35925 requested a full tablet report, sending...
I20260925 04:42:58.356392 20133 ts_manager.cc:194] Registered new tserver with Master: 9a38b2f488de4d7d8a6b26c34c737466 (127.0.98.3:37179)
I20260925 04:42:58.356988 20133 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:38579
I20260925 04:42:58.358925 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:42:58.364576 20172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:58.360955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58354 (local address 127.0.98.62:35925)
0925 04:42:58.361088 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:58.361092 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:58.361244 (+ 152us) server_negotiation.cc:407] Connection header received
0925 04:42:58.361345 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:58.361348 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:58.361390 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:58.361456 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:42:58.361666 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:58.362201 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:58.363024 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:58.363228 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:58.363300 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:58.364106 (+ 806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:42:58.364110 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:42:58.364112 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:42:58.364125 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:42:58.364143 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:42:58.364158 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:42:58.364218 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:58.364363 (+ 145us) server_negotiation.cc:299] Negotiation successful
0925 04:42:58.364424 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 04:42:58.367543 20133 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58354:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:42:58.370651 20133 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:42:58.385999 20599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:58.381086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46882 (local address 127.0.98.3:37179)
0925 04:42:58.381446 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:58.381456 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:58.381477 (+ 21us) server_negotiation.cc:407] Connection header received
0925 04:42:58.381828 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:58.381834 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:58.382006 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:58.382156 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:42:58.382376 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:58.383376 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:58.384525 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:58.385267 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:58.385374 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:58.385582 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:58.385724 (+ 142us) server_negotiation.cc:299] Negotiation successful
0925 04:42:58.385827 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":86,"threads_started":1}
I20260925 04:42:58.385993 20594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:58.380711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37179 (local address 127.0.0.1:46882)
0925 04:42:58.380950 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:58.380960 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 04:42:58.381032 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:42:58.382147 (+ 1115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:42:58.382151 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:42:58.382170 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:42:58.382250 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:58.382255 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:58.383486 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:58.383489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:58.384400 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:58.384408 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:58.385393 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:58.385397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:58.385512 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:58.385552 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 04:42:58.385677 (+ 125us) client_negotiation.cc:240] Negotiation successful
0925 04:42:58.385823 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":86,"threads_started":1}
I20260925 04:42:58.386227 20597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:58.380403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53778 (local address 127.0.98.1:42551)
0925 04:42:58.380725 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:58.380736 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:58.381031 (+ 295us) server_negotiation.cc:407] Connection header received
0925 04:42:58.381216 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:58.381220 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:58.381422 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:58.381543 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:42:58.381902 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:58.382782 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:58.384476 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:58.385300 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:58.385381 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:58.385581 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:58.385955 (+ 374us) server_negotiation.cc:299] Negotiation successful
0925 04:42:58.386058 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":124,"threads_started":1}
I20260925 04:42:58.386353 20595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:58.379917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42505 (local address 127.0.0.1:43474)
0925 04:42:58.380564 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:58.380618 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 04:42:58.380752 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:42:58.381554 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:42:58.381560 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:42:58.381581 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:42:58.381722 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:58.381729 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:58.383149 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:58.383152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:58.384344 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:58.384351 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:58.385647 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:58.385651 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:58.385789 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:58.385832 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 04:42:58.385889 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 04:42:58.385965 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"mutex_wait_us":84,"thread_start_us":111,"threads_started":1}
I20260925 04:42:58.386610 20598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:58.380026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42551 (local address 127.0.0.1:53778)
0925 04:42:58.380937 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:58.380949 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:42:58.381087 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:42:58.381683 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:42:58.381685 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:42:58.381700 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:42:58.381784 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:58.381789 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:58.382891 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:58.382893 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:58.384332 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:58.384344 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:58.385665 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:58.385667 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:58.385780 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:58.385826 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 04:42:58.386405 (+ 579us) client_negotiation.cc:240] Negotiation successful
0925 04:42:58.386489 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":849,"mutex_wait_us":13,"thread_start_us":102,"threads_started":1}
I20260925 04:42:58.386893 20596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:58.380055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43474 (local address 127.0.98.2:42505)
0925 04:42:58.380854 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:58.380865 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:58.380893 (+ 28us) server_negotiation.cc:407] Connection header received
0925 04:42:58.380975 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:58.380980 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:58.381215 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:58.381375 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:42:58.381854 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:58.383033 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:58.384475 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:58.385297 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:58.385396 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:58.385581 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:58.386600 (+ 1019us) server_negotiation.cc:299] Negotiation successful
0925 04:42:58.386721 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":690,"thread_start_us":109,"threads_started":1}
I20260925 04:42:58.389477 20386 tablet_service.cc:1467] Processing CreateTablet for tablet 1cbfab49914749888c56e2621e1ed687 (DEFAULT_TABLE table=test-table [id=012dd6926d864ba79865c647346c5826]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:42:58.389547 20517 tablet_service.cc:1467] Processing CreateTablet for tablet 1cbfab49914749888c56e2621e1ed687 (DEFAULT_TABLE table=test-table [id=012dd6926d864ba79865c647346c5826]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:42:58.389547 20255 tablet_service.cc:1467] Processing CreateTablet for tablet 1cbfab49914749888c56e2621e1ed687 (DEFAULT_TABLE table=test-table [id=012dd6926d864ba79865c647346c5826]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:42:58.390566 20386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cbfab49914749888c56e2621e1ed687. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:42:58.390614 20255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cbfab49914749888c56e2621e1ed687. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:42:58.390614 20517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cbfab49914749888c56e2621e1ed687. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:42:58.397918 20600 tablet_bootstrap.cc:492] T 1cbfab49914749888c56e2621e1ed687 P fd00ae08be8a4b13bd7c60ea0d45cb49: Bootstrap starting.
I20260925 04:42:58.398639 20601 tablet_bootstrap.cc:492] T 1cbfab49914749888c56e2621e1ed687 P 9a38b2f488de4d7d8a6b26c34c737466: Bootstrap starting.
I20260925 04:42:58.399914 20600 tablet_bootstrap.cc:654] T 1cbfab49914749888c56e2621e1ed687 P fd00ae08be8a4b13bd7c60ea0d45cb49: Neither blocks nor log segments found. Creating new log.
I20260925 04:42:58.399953 20602 tablet_bootstrap.cc:492] T 1cbfab49914749888c56e2621e1ed687 P 3b6cbdad86bc46a1a691ba97b7fb18f8: Bootstrap starting.
I20260925 04:42:58.400527 20601 tablet_bootstrap.cc:654] T 1cbfab49914749888c56e2621e1ed687 P 9a38b2f488de4d7d8a6b26c34c737466: Neither blocks nor log segments found. Creating new log.
I20260925 04:42:58.400687 20600 log.cc:824] T 1cbfab49914749888c56e2621e1ed687 P fd00ae08be8a4b13bd7c60ea0d45cb49: Log is configured to *not* fsync() on all Append() calls
I20260925 04:42:58.401384 20601 log.cc:824] T 1cbfab49914749888c56e2621e1ed687 P 9a38b2f488de4d7d8a6b26c34c737466: Log is configured to *not* fsync() on all Append() calls
I20260925 04:42:58.402094 20602 tablet_bootstrap.cc:654] T 1cbfab49914749888c56e2621e1ed687 P 3b6cbdad86bc46a1a691ba97b7fb18f8: Neither blocks nor log segments found. Creating new log.
I20260925 04:42:58.402839 20602 log.cc:824] T 1cbfab49914749888c56e2621e1ed687 P 3b6cbdad86bc46a1a691ba97b7fb18f8: Log is configured to *not* fsync() on all Append() calls
I20260925 04:42:58.403086 20600 tablet_bootstrap.cc:492] T 1cbfab49914749888c56e2621e1ed687 P fd00ae08be8a4b13bd7c60ea0d45cb49: No bootstrap required, opened a new log
I20260925 04:42:58.403292 20600 ts_tablet_manager.cc:1397] T 1cbfab49914749888c56e2621e1ed687 P fd00ae08be8a4b13bd7c60ea0d45cb49: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260925 04:42:58.404830 20601 tablet_bootstrap.cc:492] T 1cbfab49914749888c56e2621e1ed687 P 9a38b2f488de4d7d8a6b26c34c737466: No bootstrap required, opened a new log
I20260925 04:42:58.404830 20602 tablet_bootstrap.cc:492] T 1cbfab49914749888c56e2621e1ed687 P 3b6cbdad86bc46a1a691ba97b7fb18f8: No bootstrap required, opened a new log
I20260925 04:42:58.405020 20602 ts_tablet_manager.cc:1397] T 1cbfab49914749888c56e2621e1ed687 P 3b6cbdad86bc46a1a691ba97b7fb18f8: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 04:42:58.405074 20601 ts_tablet_manager.cc:1397] T 1cbfab49914749888c56e2621e1ed687 P 9a38b2f488de4d7d8a6b26c34c737466: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 04:42:58.406342 20600 raft_consensus.cc:348] T 1cbfab49914749888c56e2621e1ed687 P fd00ae08be8a4b13bd7c60ea0d45cb49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd00ae08be8a4b13bd7c60ea0d45cb49" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } } peers { permanent_uuid: "3b6cbdad86bc46a1a691ba97b7fb18f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42551 } } peers { permanent_uuid: "9a38b2f488de4d7d8a6b26c34c737466" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37179 } }
I20260925 04:42:58.406566 20600 raft_consensus.cc:374] T 1cbfab49914749888c56e2621e1ed687 P fd00ae08be8a4b13bd7c60ea0d45cb49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:42:58.406616 20600 raft_consensus.cc:729] T 1cbfab49914749888c56e2621e1ed687 P fd00ae08be8a4b13bd7c60ea0d45cb49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd00ae08be8a4b13bd7c60ea0d45cb49, State: Initialized, Role: FOLLOWER
I20260925 04:42:58.407035 20600 consensus_queue.cc:260] T 1cbfab49914749888c56e2621e1ed687 P fd00ae08be8a4b13bd7c60ea0d45cb49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd00ae08be8a4b13bd7c60ea0d45cb49" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } } peers { permanent_uuid: "3b6cbdad86bc46a1a691ba97b7fb18f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42551 } } peers { permanent_uuid: "9a38b2f488de4d7d8a6b26c34c737466" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37179 } }
I20260925 04:42:58.407723 20452 heartbeater.cc:503] Master 127.0.98.62:35925 was elected leader, sending a full tablet report...
I20260925 04:42:58.408072 20600 ts_tablet_manager.cc:1428] T 1cbfab49914749888c56e2621e1ed687 P fd00ae08be8a4b13bd7c60ea0d45cb49: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 04:42:58.409179 20602 raft_consensus.cc:348] T 1cbfab49914749888c56e2621e1ed687 P 3b6cbdad86bc46a1a691ba97b7fb18f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd00ae08be8a4b13bd7c60ea0d45cb49" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } } peers { permanent_uuid: "3b6cbdad86bc46a1a691ba97b7fb18f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42551 } } peers { permanent_uuid: "9a38b2f488de4d7d8a6b26c34c737466" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37179 } }
I20260925 04:42:58.409442 20602 raft_consensus.cc:374] T 1cbfab49914749888c56e2621e1ed687 P 3b6cbdad86bc46a1a691ba97b7fb18f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:42:58.409500 20602 raft_consensus.cc:729] T 1cbfab49914749888c56e2621e1ed687 P 3b6cbdad86bc46a1a691ba97b7fb18f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b6cbdad86bc46a1a691ba97b7fb18f8, State: Initialized, Role: FOLLOWER
I20260925 04:42:58.409767 20601 raft_consensus.cc:348] T 1cbfab49914749888c56e2621e1ed687 P 9a38b2f488de4d7d8a6b26c34c737466 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd00ae08be8a4b13bd7c60ea0d45cb49" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } } peers { permanent_uuid: "3b6cbdad86bc46a1a691ba97b7fb18f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42551 } } peers { permanent_uuid: "9a38b2f488de4d7d8a6b26c34c737466" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37179 } }
I20260925 04:42:58.409969 20601 raft_consensus.cc:374] T 1cbfab49914749888c56e2621e1ed687 P 9a38b2f488de4d7d8a6b26c34c737466 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:42:58.410018 20601 raft_consensus.cc:729] T 1cbfab49914749888c56e2621e1ed687 P 9a38b2f488de4d7d8a6b26c34c737466 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a38b2f488de4d7d8a6b26c34c737466, State: Initialized, Role: FOLLOWER
I20260925 04:42:58.409953 20602 consensus_queue.cc:260] T 1cbfab49914749888c56e2621e1ed687 P 3b6cbdad86bc46a1a691ba97b7fb18f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd00ae08be8a4b13bd7c60ea0d45cb49" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } } peers { permanent_uuid: "3b6cbdad86bc46a1a691ba97b7fb18f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42551 } } peers { permanent_uuid: "9a38b2f488de4d7d8a6b26c34c737466" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37179 } }
I20260925 04:42:58.410405 20601 consensus_queue.cc:260] T 1cbfab49914749888c56e2621e1ed687 P 9a38b2f488de4d7d8a6b26c34c737466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd00ae08be8a4b13bd7c60ea0d45cb49" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } } peers { permanent_uuid: "3b6cbdad86bc46a1a691ba97b7fb18f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42551 } } peers { permanent_uuid: "9a38b2f488de4d7d8a6b26c34c737466" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37179 } }
I20260925 04:42:58.411001 20602 ts_tablet_manager.cc:1428] T 1cbfab49914749888c56e2621e1ed687 P 3b6cbdad86bc46a1a691ba97b7fb18f8: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 04:42:58.411059 20583 heartbeater.cc:503] Master 127.0.98.62:35925 was elected leader, sending a full tablet report...
I20260925 04:42:58.411310 20601 ts_tablet_manager.cc:1428] T 1cbfab49914749888c56e2621e1ed687 P 9a38b2f488de4d7d8a6b26c34c737466: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260925 04:42:58.411309 20321 heartbeater.cc:503] Master 127.0.98.62:35925 was elected leader, sending a full tablet report...
W20260925 04:42:58.459826 20322 tablet.cc:2369] T 1cbfab49914749888c56e2621e1ed687 P 3b6cbdad86bc46a1a691ba97b7fb18f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 04:42:58.593842 20584 tablet.cc:2369] T 1cbfab49914749888c56e2621e1ed687 P 9a38b2f488de4d7d8a6b26c34c737466: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 04:42:58.650817 20453 tablet.cc:2369] T 1cbfab49914749888c56e2621e1ed687 P fd00ae08be8a4b13bd7c60ea0d45cb49: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:42:58.798343 20607 raft_consensus.cc:482] T 1cbfab49914749888c56e2621e1ed687 P 3b6cbdad86bc46a1a691ba97b7fb18f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:42:58.798614 20607 raft_consensus.cc:504] T 1cbfab49914749888c56e2621e1ed687 P 3b6cbdad86bc46a1a691ba97b7fb18f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd00ae08be8a4b13bd7c60ea0d45cb49" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } } peers { permanent_uuid: "3b6cbdad86bc46a1a691ba97b7fb18f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42551 } } peers { permanent_uuid: "9a38b2f488de4d7d8a6b26c34c737466" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37179 } }
I20260925 04:42:58.799739 20607 leader_election.cc:288] T 1cbfab49914749888c56e2621e1ed687 P 3b6cbdad86bc46a1a691ba97b7fb18f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd00ae08be8a4b13bd7c60ea0d45cb49 (127.0.98.2:42505), 9a38b2f488de4d7d8a6b26c34c737466 (127.0.98.3:37179)
I20260925 04:42:58.804123 20609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:58.800012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42505 (local address 127.0.98.1:34443)
0925 04:42:58.800268 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:58.800282 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:42:58.800410 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:42:58.800717 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:42:58.800720 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:42:58.800742 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:42:58.800857 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:58.800863 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:58.801758 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:58.801761 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:58.802833 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:58.802853 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:58.803586 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:58.803591 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:58.803713 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:58.803742 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 04:42:58.803832 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 04:42:58.803915 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":73,"threads_started":1}
I20260925 04:42:58.804198 20596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:58.800122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34443 (local address 127.0.98.2:42505)
0925 04:42:58.800282 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:58.800287 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:58.800370 (+ 83us) server_negotiation.cc:407] Connection header received
0925 04:42:58.800547 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:58.800551 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:58.800608 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:58.800690 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:42:58.800999 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:58.801625 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:58.802990 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:58.803446 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:58.803530 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:58.803585 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:58.803965 (+ 380us) server_negotiation.cc:299] Negotiation successful
0925 04:42:58.804069 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 04:42:58.804400 20610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:58.800390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37179 (local address 127.0.98.1:54713)
0925 04:42:58.800754 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:58.800767 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:42:58.800852 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:42:58.801104 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:42:58.801107 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:42:58.801125 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:42:58.801248 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:58.801254 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:58.802256 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:58.802259 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:58.803068 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:42:58.803078 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:42:58.803981 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:42:58.803985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:42:58.804087 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:58.804109 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 04:42:58.804187 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 04:42:58.804256 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":81,"threads_started":1}
I20260925 04:42:58.804566 20599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:58.800524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54713 (local address 127.0.98.3:37179)
0925 04:42:58.800648 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:58.800651 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:58.800845 (+ 194us) server_negotiation.cc:407] Connection header received
0925 04:42:58.800961 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:58.800965 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:58.801013 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:58.801069 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:42:58.801466 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:58.802138 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:58.803217 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:58.803854 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:58.803950 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:58.804015 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:58.804353 (+ 338us) server_negotiation.cc:299] Negotiation successful
0925 04:42:58.804445 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 04:42:58.804978 20406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cbfab49914749888c56e2621e1ed687" candidate_uuid: "3b6cbdad86bc46a1a691ba97b7fb18f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd00ae08be8a4b13bd7c60ea0d45cb49" is_pre_election: true
I20260925 04:42:58.805058 20537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cbfab49914749888c56e2621e1ed687" candidate_uuid: "3b6cbdad86bc46a1a691ba97b7fb18f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a38b2f488de4d7d8a6b26c34c737466" is_pre_election: true
I20260925 04:42:58.805325 20537 raft_consensus.cc:2454] T 1cbfab49914749888c56e2621e1ed687 P 9a38b2f488de4d7d8a6b26c34c737466 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b6cbdad86bc46a1a691ba97b7fb18f8 in term 0.
I20260925 04:42:58.805326 20406 raft_consensus.cc:2454] T 1cbfab49914749888c56e2621e1ed687 P fd00ae08be8a4b13bd7c60ea0d45cb49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b6cbdad86bc46a1a691ba97b7fb18f8 in term 0.
I20260925 04:42:58.805843 20208 leader_election.cc:302] T 1cbfab49914749888c56e2621e1ed687 P 3b6cbdad86bc46a1a691ba97b7fb18f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b6cbdad86bc46a1a691ba97b7fb18f8, 9a38b2f488de4d7d8a6b26c34c737466; no voters:
I20260925 04:42:58.806121 20607 raft_consensus.cc:2781] T 1cbfab49914749888c56e2621e1ed687 P 3b6cbdad86bc46a1a691ba97b7fb18f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:42:58.806212 20607 raft_consensus.cc:482] T 1cbfab49914749888c56e2621e1ed687 P 3b6cbdad86bc46a1a691ba97b7fb18f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:42:58.806255 20607 raft_consensus.cc:3037] T 1cbfab49914749888c56e2621e1ed687 P 3b6cbdad86bc46a1a691ba97b7fb18f8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:42:58.807325 20607 raft_consensus.cc:504] T 1cbfab49914749888c56e2621e1ed687 P 3b6cbdad86bc46a1a691ba97b7fb18f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd00ae08be8a4b13bd7c60ea0d45cb49" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } } peers { permanent_uuid: "3b6cbdad86bc46a1a691ba97b7fb18f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42551 } } peers { permanent_uuid: "9a38b2f488de4d7d8a6b26c34c737466" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37179 } }
I20260925 04:42:58.807713 20607 leader_election.cc:288] T 1cbfab49914749888c56e2621e1ed687 P 3b6cbdad86bc46a1a691ba97b7fb18f8 [CANDIDATE]: Term 1 election: Requested vote from peers fd00ae08be8a4b13bd7c60ea0d45cb49 (127.0.98.2:42505), 9a38b2f488de4d7d8a6b26c34c737466 (127.0.98.3:37179)
I20260925 04:42:58.808132 20406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cbfab49914749888c56e2621e1ed687" candidate_uuid: "3b6cbdad86bc46a1a691ba97b7fb18f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd00ae08be8a4b13bd7c60ea0d45cb49"
I20260925 04:42:58.808166 20537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cbfab49914749888c56e2621e1ed687" candidate_uuid: "3b6cbdad86bc46a1a691ba97b7fb18f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a38b2f488de4d7d8a6b26c34c737466"
I20260925 04:42:58.808292 20537 raft_consensus.cc:3037] T 1cbfab49914749888c56e2621e1ed687 P 9a38b2f488de4d7d8a6b26c34c737466 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:42:58.808269 20406 raft_consensus.cc:3037] T 1cbfab49914749888c56e2621e1ed687 P fd00ae08be8a4b13bd7c60ea0d45cb49 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:42:58.809407 20406 raft_consensus.cc:2454] T 1cbfab49914749888c56e2621e1ed687 P fd00ae08be8a4b13bd7c60ea0d45cb49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b6cbdad86bc46a1a691ba97b7fb18f8 in term 1.
I20260925 04:42:58.809567 20537 raft_consensus.cc:2454] T 1cbfab49914749888c56e2621e1ed687 P 9a38b2f488de4d7d8a6b26c34c737466 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b6cbdad86bc46a1a691ba97b7fb18f8 in term 1.
I20260925 04:42:58.809845 20208 leader_election.cc:302] T 1cbfab49914749888c56e2621e1ed687 P 3b6cbdad86bc46a1a691ba97b7fb18f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b6cbdad86bc46a1a691ba97b7fb18f8, fd00ae08be8a4b13bd7c60ea0d45cb49; no voters:
I20260925 04:42:58.810034 20607 raft_consensus.cc:2781] T 1cbfab49914749888c56e2621e1ed687 P 3b6cbdad86bc46a1a691ba97b7fb18f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:42:58.810249 20607 raft_consensus.cc:686] T 1cbfab49914749888c56e2621e1ed687 P 3b6cbdad86bc46a1a691ba97b7fb18f8 [term 1 LEADER]: Becoming Leader. State: Replica: 3b6cbdad86bc46a1a691ba97b7fb18f8, State: Running, Role: LEADER
I20260925 04:42:58.810591 20607 consensus_queue.cc:237] T 1cbfab49914749888c56e2621e1ed687 P 3b6cbdad86bc46a1a691ba97b7fb18f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd00ae08be8a4b13bd7c60ea0d45cb49" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } } peers { permanent_uuid: "3b6cbdad86bc46a1a691ba97b7fb18f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42551 } } peers { permanent_uuid: "9a38b2f488de4d7d8a6b26c34c737466" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37179 } }
I20260925 04:42:58.814160 20133 catalog_manager.cc:5495] T 1cbfab49914749888c56e2621e1ed687 P 3b6cbdad86bc46a1a691ba97b7fb18f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b6cbdad86bc46a1a691ba97b7fb18f8 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "3b6cbdad86bc46a1a691ba97b7fb18f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd00ae08be8a4b13bd7c60ea0d45cb49" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b6cbdad86bc46a1a691ba97b7fb18f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a38b2f488de4d7d8a6b26c34c737466" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37179 } health_report { overall_health: UNKNOWN } } }
I20260925 04:42:58.844209 20597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:58.840710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46650 (local address 127.0.98.1:42551)
0925 04:42:58.840832 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 04:42:58.840836 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:42:58.840848 (+ 12us) server_negotiation.cc:407] Connection header received
0925 04:42:58.840949 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:42:58.840951 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:42:58.841005 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:42:58.841112 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:42:58.841347 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:58.841930 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:58.842765 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:42:58.842976 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:42:58.843063 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:42:58.843369 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:42:58.843820 (+ 451us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:42:58.843925 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 04:42:58.844003 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 04:42:58.844067 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 04:42:58.850531 20406 raft_consensus.cc:1261] T 1cbfab49914749888c56e2621e1ed687 P fd00ae08be8a4b13bd7c60ea0d45cb49 [term 1 FOLLOWER]: Refusing update from remote peer 3b6cbdad86bc46a1a691ba97b7fb18f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:42:58.850538 20537 raft_consensus.cc:1261] T 1cbfab49914749888c56e2621e1ed687 P 9a38b2f488de4d7d8a6b26c34c737466 [term 1 FOLLOWER]: Refusing update from remote peer 3b6cbdad86bc46a1a691ba97b7fb18f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:42:58.851241 20611 consensus_queue.cc:1035] T 1cbfab49914749888c56e2621e1ed687 P 3b6cbdad86bc46a1a691ba97b7fb18f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd00ae08be8a4b13bd7c60ea0d45cb49" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:42:58.851442 20607 consensus_queue.cc:1035] T 1cbfab49914749888c56e2621e1ed687 P 3b6cbdad86bc46a1a691ba97b7fb18f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a38b2f488de4d7d8a6b26c34c737466" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:42:58.861435 20615 mvcc.cc:205] Tried to move back new op lower bound from 7333115407764160512 to 7333115407611564032. Current Snapshot: MvccSnapshot[applied={T|T < 7333115407764160512}]
I20260925 04:42:58.867406 20133 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58354:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:42:58.867574 20133 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58354:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:42:58.870055 20133 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 05f61fab032f495c8faa7dd631f17594: Sending DeleteTablet for 3 replicas of tablet 1cbfab49914749888c56e2621e1ed687
I20260925 04:42:58.871006 20386 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cbfab49914749888c56e2621e1ed687 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:42:58 UTC) from {username='slave'} at 127.0.0.1:43474
I20260925 04:42:58.871111 20255 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cbfab49914749888c56e2621e1ed687 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:42:58 UTC) from {username='slave'} at 127.0.0.1:53778
I20260925 04:42:58.871479 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 20193
I20260925 04:42:58.871644 20517 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cbfab49914749888c56e2621e1ed687 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:42:58 UTC) from {username='slave'} at 127.0.0.1:46882
I20260925 04:42:58.872687 20624 tablet_replica.cc:330] T 1cbfab49914749888c56e2621e1ed687 P fd00ae08be8a4b13bd7c60ea0d45cb49: stopping tablet replica
I20260925 04:42:58.873042 20624 raft_consensus.cc:2229] T 1cbfab49914749888c56e2621e1ed687 P fd00ae08be8a4b13bd7c60ea0d45cb49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:42:58.873183 20626 tablet_replica.cc:330] T 1cbfab49914749888c56e2621e1ed687 P 9a38b2f488de4d7d8a6b26c34c737466: stopping tablet replica
I20260925 04:42:58.873417 20624 raft_consensus.cc:2258] T 1cbfab49914749888c56e2621e1ed687 P fd00ae08be8a4b13bd7c60ea0d45cb49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:42:58.873507 20626 raft_consensus.cc:2229] T 1cbfab49914749888c56e2621e1ed687 P 9a38b2f488de4d7d8a6b26c34c737466 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:42:58.873819 20626 raft_consensus.cc:2258] T 1cbfab49914749888c56e2621e1ed687 P 9a38b2f488de4d7d8a6b26c34c737466 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:42:58.874811 20624 ts_tablet_manager.cc:1905] T 1cbfab49914749888c56e2621e1ed687 P fd00ae08be8a4b13bd7c60ea0d45cb49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:42:58.876200 20626 ts_tablet_manager.cc:1905] T 1cbfab49914749888c56e2621e1ed687 P 9a38b2f488de4d7d8a6b26c34c737466: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:42:58.877677 20624 ts_tablet_manager.cc:1918] T 1cbfab49914749888c56e2621e1ed687 P fd00ae08be8a4b13bd7c60ea0d45cb49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:42:58.877782 20624 log.cc:1196] T 1cbfab49914749888c56e2621e1ed687 P fd00ae08be8a4b13bd7c60ea0d45cb49: Deleting WAL directory at /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311123478858-392-0/minicluster-data/ts-1/wal/wals/1cbfab49914749888c56e2621e1ed687
I20260925 04:42:58.878085 20624 ts_tablet_manager.cc:1939] T 1cbfab49914749888c56e2621e1ed687 P fd00ae08be8a4b13bd7c60ea0d45cb49: Deleting consensus metadata
W20260925 04:42:58.878072 20118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42551 (error 108)
I20260925 04:42:58.878286 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 20324
I20260925 04:42:58.879532 20626 ts_tablet_manager.cc:1918] T 1cbfab49914749888c56e2621e1ed687 P 9a38b2f488de4d7d8a6b26c34c737466: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:42:58.879647 20626 log.cc:1196] T 1cbfab49914749888c56e2621e1ed687 P 9a38b2f488de4d7d8a6b26c34c737466: Deleting WAL directory at /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311123478858-392-0/minicluster-data/ts-2/wal/wals/1cbfab49914749888c56e2621e1ed687
I20260925 04:42:58.879909 20626 ts_tablet_manager.cc:1939] T 1cbfab49914749888c56e2621e1ed687 P 9a38b2f488de4d7d8a6b26c34c737466: Deleting consensus metadata
I20260925 04:42:58.881155 20117 catalog_manager.cc:4841] TS 9a38b2f488de4d7d8a6b26c34c737466 (127.0.98.3:37179): tablet 1cbfab49914749888c56e2621e1ed687 (table test-table [id=012dd6926d864ba79865c647346c5826]) successfully deleted
I20260925 04:42:58.884182 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 20455
I20260925 04:42:58.884524 20598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:58.884220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42551 (local address 127.0.0.1:46652)
0925 04:42:58.884359 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:58.884433 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 04:42:58.884721 20598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:42:58.884288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42505 (local address 127.0.0.1:38516)
0925 04:42:58.884645 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0925 04:42:58.884681 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316}
W20260925 04:42:58.884855 20118 catalog_manager.cc:4584] TS 3b6cbdad86bc46a1a691ba97b7fb18f8 (127.0.98.1:42551): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1cbfab49914749888c56e2621e1ed687: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42551: connect: Connection refused (error 111)
I20260925 04:42:58.889449 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 20102
2026-09-25T04:42:58Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1377 ms)
[----------] 6 tests from EncryptionPolicyTest (10401 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790311123478858-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:42:58 dist-test-slave-dc95 krb5kdc[20630](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:42:58 dist-test-slave-dc95 krb5kdc[20630](info): set up 2 sockets
Sep 25 04:42:58 dist-test-slave-dc95 krb5kdc[20630](info): commencing operation
krb5kdc: starting...
W20260925 04:42:59.847627 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790311123478858-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790311123478858-392-0/krb5kdc/test-user.keytab.
Sep 25 04:42:59 dist-test-slave-dc95 krb5kdc[20630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:42:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:42:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790311123478858-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790311123478858-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 04:42:59.967687 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790311123478858-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790311123478858-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790311123478858-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43179
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790311123478858-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43179
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:43:00.081725 20643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:43:00.082016 20643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:43:00.082060 20643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:43:00.086169 20643 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:43:00.086246 20643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:43:00.086264 20643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:43:00.086280 20643 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:43:00.086294 20643 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:43:00.092213 20643 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790311123478858-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790311123478858-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43179
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790311123478858-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790311123478858-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:43:00.093289 20643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:43:00.094264 20643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:43:00.102922 20649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:43:00.103003 20643 server_base.cc:1034] running on GCE node
W20260925 04:43:00.102946 20648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:43:00.102921 20651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:43:00.103891 20643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:43:00.105078 20643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:43:00.106298 20643 hybrid_clock.cc:648] HybridClock initialized: now 1790311380106275 us; error 34 us; skew 500 ppm
Sep 25 04:43:00 dist-test-slave-dc95 krb5kdc[20630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:43:00.109273 20643 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 04:43:00.110244 20643 webserver.cc:460] Webserver started at http://127.0.0.1:36749/ using document root <none> and password file <none>
I20260925 04:43:00.110780 20643 fs_manager.cc:359] Metadata directory not provided
I20260925 04:43:00.110838 20643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:43:00.111001 20643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:43:00.113145 20643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790311123478858-392-0/minicluster-data/master-0/data/instance:
uuid: "a9cdfc92421047e99b217893fdfd9d0e"
format_stamp: "Formatted at 2026-09-25 04:43:00 on dist-test-slave-dc95"
server_key: "18e7b0523c8f70a25872a818280cc385"
server_key_iv: "3a1b21e034c076a42f3c4d0eb8e77b72"
server_key_version: "encryptionkey@0"
I20260925 04:43:00.113673 20643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790311123478858-392-0/minicluster-data/master-0/wal/instance:
uuid: "a9cdfc92421047e99b217893fdfd9d0e"
format_stamp: "Formatted at 2026-09-25 04:43:00 on dist-test-slave-dc95"
server_key: "18e7b0523c8f70a25872a818280cc385"
server_key_iv: "3a1b21e034c076a42f3c4d0eb8e77b72"
server_key_version: "encryptionkey@0"
I20260925 04:43:00.117693 20643 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 04:43:00.120329 20658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:43:00.121706 20643 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:43:00.121897 20643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790311123478858-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790311123478858-392-0/minicluster-data/master-0/wal
uuid: "a9cdfc92421047e99b217893fdfd9d0e"
format_stamp: "Formatted at 2026-09-25 04:43:00 on dist-test-slave-dc95"
server_key: "18e7b0523c8f70a25872a818280cc385"
server_key_iv: "3a1b21e034c076a42f3c4d0eb8e77b72"
server_key_version: "encryptionkey@0"
I20260925 04:43:00.122021 20643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790311123478858-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790311123478858-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790311123478858-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:43:00.134277 20643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:43:00.137310 20643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:43:00.137490 20643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:43:00.144681 20643 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43179
I20260925 04:43:00.144701 20710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43179 every 8 connection(s)
I20260925 04:43:00.145696 20643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
I20260925 04:43:00.148588 20711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:43:00.153946 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 20643
I20260925 04:43:00.154088 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790311123478858-392-0/minicluster-data/master-0/wal/instance
I20260925 04:43:00.154336 392 external_mini_cluster.cc:1428] Setting key 32cd9a7816a55a88725882320226e9af
I20260925 04:43:00.154450 20711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9cdfc92421047e99b217893fdfd9d0e: Bootstrap starting.
I20260925 04:43:00.157159 20711 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9cdfc92421047e99b217893fdfd9d0e: Neither blocks nor log segments found. Creating new log.
I20260925 04:43:00.158043 20711 log.cc:824] T 00000000000000000000000000000000 P a9cdfc92421047e99b217893fdfd9d0e: Log is configured to *not* fsync() on all Append() calls
I20260925 04:43:00.160121 20711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9cdfc92421047e99b217893fdfd9d0e: No bootstrap required, opened a new log
Sep 25 04:43:00 dist-test-slave-dc95 krb5kdc[20630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 04:43:00.163777 20711 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9cdfc92421047e99b217893fdfd9d0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9cdfc92421047e99b217893fdfd9d0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43179 } }
I20260925 04:43:00.164039 20711 raft_consensus.cc:374] T 00000000000000000000000000000000 P a9cdfc92421047e99b217893fdfd9d0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:43:00.164099 20711 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9cdfc92421047e99b217893fdfd9d0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9cdfc92421047e99b217893fdfd9d0e, State: Initialized, Role: FOLLOWER
I20260925 04:43:00.164551 20711 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9cdfc92421047e99b217893fdfd9d0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9cdfc92421047e99b217893fdfd9d0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43179 } }
I20260925 04:43:00.164696 20711 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9cdfc92421047e99b217893fdfd9d0e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:43:00.164765 20711 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9cdfc92421047e99b217893fdfd9d0e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:43:00.164872 20711 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9cdfc92421047e99b217893fdfd9d0e [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:43:00.166050 20711 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9cdfc92421047e99b217893fdfd9d0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9cdfc92421047e99b217893fdfd9d0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43179 } }
I20260925 04:43:00.166389 20711 leader_election.cc:302] T 00000000000000000000000000000000 P a9cdfc92421047e99b217893fdfd9d0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9cdfc92421047e99b217893fdfd9d0e; no voters:
I20260925 04:43:00.166730 20711 leader_election.cc:288] T 00000000000000000000000000000000 P a9cdfc92421047e99b217893fdfd9d0e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:43:00.166852 20716 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9cdfc92421047e99b217893fdfd9d0e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:43:00.167086 20716 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9cdfc92421047e99b217893fdfd9d0e [term 1 LEADER]: Becoming Leader. State: Replica: a9cdfc92421047e99b217893fdfd9d0e, State: Running, Role: LEADER
I20260925 04:43:00.167430 20716 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9cdfc92421047e99b217893fdfd9d0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9cdfc92421047e99b217893fdfd9d0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43179 } }
I20260925 04:43:00.168200 20711 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9cdfc92421047e99b217893fdfd9d0e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:43:00.169253 20717 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9cdfc92421047e99b217893fdfd9d0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a9cdfc92421047e99b217893fdfd9d0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9cdfc92421047e99b217893fdfd9d0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43179 } } }
I20260925 04:43:00.169389 20717 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9cdfc92421047e99b217893fdfd9d0e [sys.catalog]: This master's current role is: LEADER
I20260925 04:43:00.169689 20718 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9cdfc92421047e99b217893fdfd9d0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9cdfc92421047e99b217893fdfd9d0e. Latest consensus state: current_term: 1 leader_uuid: "a9cdfc92421047e99b217893fdfd9d0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9cdfc92421047e99b217893fdfd9d0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43179 } } }
I20260925 04:43:00.169777 20718 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9cdfc92421047e99b217893fdfd9d0e [sys.catalog]: This master's current role is: LEADER
I20260925 04:43:00.169975 20714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:00.155726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49299 (local address 127.0.0.1:43179)
0925 04:43:00.156197 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:00.156213 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:00.156275 (+ 62us) server_negotiation.cc:407] Connection header received
0925 04:43:00.156989 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:00.157009 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:00.157430 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:00.157804 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:00.158657 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:00.159737 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:00.160659 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:00.161150 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:00.161319 (+ 169us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:00.163763 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:43:00.163769 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:43:00.163785 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:43:00.163815 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:43:00.165746 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:00.166323 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:00.166329 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:00.166334 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:00.166452 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:00.166960 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:00.166963 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:00.166966 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:00.167502 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:43:00.167732 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:00.168255 (+ 523us) server_negotiation.cc:299] Negotiation successful
0925 04:43:00.168633 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":147,"threads_started":1}
I20260925 04:43:00.170058 20725 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:43:00.173233 20725 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:43:00.173640 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:43:00.179284 20725 catalog_manager.cc:1347] Generated new cluster ID: e080ff8f935c4607af61c512322d2fad
I20260925 04:43:00.179360 20725 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:43:00.183864 20714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:00.175815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33299 (local address 127.0.0.1:43179)
0925 04:43:00.175932 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:00.175936 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:00.176174 (+ 238us) server_negotiation.cc:407] Connection header received
0925 04:43:00.176280 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:00.176283 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:00.176327 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:00.176401 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:00.177032 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:00.177654 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:00.178805 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:00.179154 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:00.179249 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:00.180776 (+ 1527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:43:00.180780 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:43:00.180783 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:43:00.180809 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:43:00.182074 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:00.182659 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:00.182662 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:00.182663 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:00.182733 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:00.183030 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:00.183033 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:00.183035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:00.183198 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:43:00.183324 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:00.183575 (+ 251us) server_negotiation.cc:299] Negotiation successful
0925 04:43:00.183700 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 04:43:00.217468 20725 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:43:00.218449 20725 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:43:00.231007 20725 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a9cdfc92421047e99b217893fdfd9d0e: Generated new TSK 0
I20260925 04:43:00.231977 20725 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:43:00.239388 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 20643
W20260925 04:43:00.244361 20628 connection.cc:535] client connection to 127.0.0.1:43179 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43179): Connection reset by peer (error 104)
2026-09-25T04:43:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1348 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790311123478858-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:43:00 dist-test-slave-dc95 krb5kdc[20742](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:43:00 dist-test-slave-dc95 krb5kdc[20742](info): set up 2 sockets
Sep 25 04:43:00 dist-test-slave-dc95 krb5kdc[20742](info): commencing operation
krb5kdc: starting...
W20260925 04:43:01.195505 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790311123478858-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790311123478858-392-0/krb5kdc/test-user.keytab.
Sep 25 04:43:01 dist-test-slave-dc95 krb5kdc[20742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:43:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:43:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790311123478858-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790311123478858-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 04:43:01.323216 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790311123478858-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790311123478858-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790311123478858-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46697
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790311123478858-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46697
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:43:01.436017 20755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:43:01.436323 20755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:43:01.436367 20755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:43:01.440500 20755 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:43:01.440574 20755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:43:01.440593 20755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:43:01.440609 20755 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:43:01.440623 20755 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:43:01.446164 20755 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790311123478858-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790311123478858-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46697
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790311123478858-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790311123478858-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:43:01.447229 20755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:43:01.448230 20755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:43:01.456974 20760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:43:01.457010 20763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:43:01.457010 20761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:43:01.457188 20755 server_base.cc:1034] running on GCE node
I20260925 04:43:01.457839 20755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:43:01.459033 20755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:43:01.460213 20755 hybrid_clock.cc:648] HybridClock initialized: now 1790311381460195 us; error 31 us; skew 500 ppm
Sep 25 04:43:01 dist-test-slave-dc95 krb5kdc[20742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311381, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:43:01.463006 20755 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 04:43:01.464102 20755 webserver.cc:460] Webserver started at http://127.0.0.1:44039/ using document root <none> and password file <none>
I20260925 04:43:01.464663 20755 fs_manager.cc:359] Metadata directory not provided
I20260925 04:43:01.464730 20755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:43:01.464937 20755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:43:01.467412 20755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790311123478858-392-0/minicluster-data/master-0/data/instance:
uuid: "76a0c704e3ae4df5bdcd4637c48aa75c"
format_stamp: "Formatted at 2026-09-25 04:43:01 on dist-test-slave-dc95"
server_key: "b7020ef2de83f9cb739f6b39f5b2403a"
server_key_iv: "c6778aabc310816a959c7a5298b8dbc6"
server_key_version: "encryptionkey@0"
I20260925 04:43:01.468112 20755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790311123478858-392-0/minicluster-data/master-0/wal/instance:
uuid: "76a0c704e3ae4df5bdcd4637c48aa75c"
format_stamp: "Formatted at 2026-09-25 04:43:01 on dist-test-slave-dc95"
server_key: "b7020ef2de83f9cb739f6b39f5b2403a"
server_key_iv: "c6778aabc310816a959c7a5298b8dbc6"
server_key_version: "encryptionkey@0"
I20260925 04:43:01.472352 20755 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:43:01.475111 20770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:43:01.476641 20755 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:43:01.476797 20755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790311123478858-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790311123478858-392-0/minicluster-data/master-0/wal
uuid: "76a0c704e3ae4df5bdcd4637c48aa75c"
format_stamp: "Formatted at 2026-09-25 04:43:01 on dist-test-slave-dc95"
server_key: "b7020ef2de83f9cb739f6b39f5b2403a"
server_key_iv: "c6778aabc310816a959c7a5298b8dbc6"
server_key_version: "encryptionkey@0"
I20260925 04:43:01.476922 20755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790311123478858-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790311123478858-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790311123478858-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:43:01.498385 20755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:43:01.501609 20755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:43:01.501824 20755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:43:01.509137 20755 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46697
I20260925 04:43:01.509145 20822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46697 every 8 connection(s)
I20260925 04:43:01.510192 20755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
I20260925 04:43:01.513167 20823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:43:01.519093 20823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76a0c704e3ae4df5bdcd4637c48aa75c: Bootstrap starting.
I20260925 04:43:01.519990 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 20755
I20260925 04:43:01.520126 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790311123478858-392-0/minicluster-data/master-0/wal/instance
I20260925 04:43:01.520455 392 external_mini_cluster.cc:1428] Setting key 9d2824d8f4a9d3e159b54113df986a10
I20260925 04:43:01.522136 20823 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76a0c704e3ae4df5bdcd4637c48aa75c: Neither blocks nor log segments found. Creating new log.
I20260925 04:43:01.523128 20823 log.cc:824] T 00000000000000000000000000000000 P 76a0c704e3ae4df5bdcd4637c48aa75c: Log is configured to *not* fsync() on all Append() calls
I20260925 04:43:01.525585 20823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76a0c704e3ae4df5bdcd4637c48aa75c: No bootstrap required, opened a new log
I20260925 04:43:01.530167 20823 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76a0c704e3ae4df5bdcd4637c48aa75c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76a0c704e3ae4df5bdcd4637c48aa75c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46697 } }
I20260925 04:43:01.530472 20823 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76a0c704e3ae4df5bdcd4637c48aa75c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:43:01.530544 20823 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76a0c704e3ae4df5bdcd4637c48aa75c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76a0c704e3ae4df5bdcd4637c48aa75c, State: Initialized, Role: FOLLOWER
I20260925 04:43:01.531059 20823 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76a0c704e3ae4df5bdcd4637c48aa75c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76a0c704e3ae4df5bdcd4637c48aa75c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46697 } }
I20260925 04:43:01.531179 20826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:01.521749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50917 (local address 127.0.0.1:46697)
0925 04:43:01.522262 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:01.522284 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:01.522334 (+ 50us) server_negotiation.cc:407] Connection header received
0925 04:43:01.523300 (+ 966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:01.523326 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:01.523817 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:01.524175 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:01.524595 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:01.526060 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:01.527007 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:01.527778 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:01.527946 (+ 168us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:01.528925 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:43:01.528931 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:43:01.528950 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:43:01.529535 (+ 585us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:43:01.529614 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:43:01.529645 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:43:01.529776 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:01.529972 (+ 196us) server_negotiation.cc:299] Negotiation successful
0925 04:43:01.530110 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":130,"threads_started":1}
I20260925 04:43:01.531360 20823 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76a0c704e3ae4df5bdcd4637c48aa75c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:43:01.531448 20823 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76a0c704e3ae4df5bdcd4637c48aa75c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:43:01.531558 20823 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76a0c704e3ae4df5bdcd4637c48aa75c [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:43:01.532665 20823 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76a0c704e3ae4df5bdcd4637c48aa75c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76a0c704e3ae4df5bdcd4637c48aa75c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46697 } }
I20260925 04:43:01.533021 20823 leader_election.cc:302] T 00000000000000000000000000000000 P 76a0c704e3ae4df5bdcd4637c48aa75c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76a0c704e3ae4df5bdcd4637c48aa75c; no voters:
I20260925 04:43:01.533327 20823 leader_election.cc:288] T 00000000000000000000000000000000 P 76a0c704e3ae4df5bdcd4637c48aa75c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:43:01.533453 20828 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76a0c704e3ae4df5bdcd4637c48aa75c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:43:01.533667 20828 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76a0c704e3ae4df5bdcd4637c48aa75c [term 1 LEADER]: Becoming Leader. State: Replica: 76a0c704e3ae4df5bdcd4637c48aa75c, State: Running, Role: LEADER
I20260925 04:43:01.533993 20828 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76a0c704e3ae4df5bdcd4637c48aa75c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76a0c704e3ae4df5bdcd4637c48aa75c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46697 } }
I20260925 04:43:01.534596 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:43:01.534679 20823 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76a0c704e3ae4df5bdcd4637c48aa75c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:43:01.536226 20830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76a0c704e3ae4df5bdcd4637c48aa75c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76a0c704e3ae4df5bdcd4637c48aa75c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76a0c704e3ae4df5bdcd4637c48aa75c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46697 } } }
I20260925 04:43:01.536376 20830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76a0c704e3ae4df5bdcd4637c48aa75c [sys.catalog]: This master's current role is: LEADER
I20260925 04:43:01.536700 20829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76a0c704e3ae4df5bdcd4637c48aa75c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76a0c704e3ae4df5bdcd4637c48aa75c. Latest consensus state: current_term: 1 leader_uuid: "76a0c704e3ae4df5bdcd4637c48aa75c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76a0c704e3ae4df5bdcd4637c48aa75c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46697 } } }
I20260925 04:43:01.536796 20829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76a0c704e3ae4df5bdcd4637c48aa75c [sys.catalog]: This master's current role is: LEADER
I20260925 04:43:01.537195 20843 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:43:01.540675 20843 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:43:01.541872 20826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:01.537673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49015 (local address 127.0.0.1:46697)
0925 04:43:01.537790 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:01.537793 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:01.538299 (+ 506us) server_negotiation.cc:407] Connection header received
0925 04:43:01.538410 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:01.538412 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:01.538454 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:01.538542 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:01.538821 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:01.539403 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:01.540271 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:01.540501 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:01.540590 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:01.541425 (+ 835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:43:01.541428 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:43:01.541430 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:43:01.541444 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:43:01.541470 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:43:01.541482 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:43:01.541556 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:01.541696 (+ 140us) server_negotiation.cc:299] Negotiation successful
0925 04:43:01.541730 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 04:43:01.546797 20843 catalog_manager.cc:1347] Generated new cluster ID: 53e765d188b144389d6164ab6b80bd6a
I20260925 04:43:01.546887 20843 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:43:01.564251 20843 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:43:01.565747 20843 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:43:01.573335 20843 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76a0c704e3ae4df5bdcd4637c48aa75c: Generated new TSK 0
I20260925 04:43:01.574105 20843 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:43:01.578352 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 20755
W20260925 04:43:01.583153 20740 connection.cc:535] client connection to 127.0.0.1:46697 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46697): Connection reset by peer (error 104)
2026-09-25T04:43:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1339 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790311123478858-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:43:01 dist-test-slave-dc95 krb5kdc[20854](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:43:01 dist-test-slave-dc95 krb5kdc[20854](info): set up 2 sockets
Sep 25 04:43:01 dist-test-slave-dc95 krb5kdc[20854](info): commencing operation
krb5kdc: starting...
W20260925 04:43:02.543495 392 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790311123478858-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790311123478858-392-0/krb5kdc/test-user.keytab.
Sep 25 04:43:02 dist-test-slave-dc95 krb5kdc[20854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:43:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:43:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790311123478858-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790311123478858-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 04:43:02.667752 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790311123478858-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790311123478858-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790311123478858-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38479
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790311123478858-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38479
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:43:02.782373 20867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:43:02.782692 20867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:43:02.782737 20867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:43:02.787235 20867 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:43:02.787309 20867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:43:02.787328 20867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:43:02.787344 20867 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:43:02.787359 20867 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:43:02.793493 20867 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790311123478858-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790311123478858-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38479
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790311123478858-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790311123478858-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:43:02.794605 20867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:43:02.795679 20867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:43:02.804612 20875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:43:02.804679 20867 server_base.cc:1034] running on GCE node
W20260925 04:43:02.804608 20873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:43:02.804605 20872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:43:02.805439 20867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:43:02.806631 20867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:43:02.807790 20867 hybrid_clock.cc:648] HybridClock initialized: now 1790311382807769 us; error 38 us; skew 500 ppm
Sep 25 04:43:02 dist-test-slave-dc95 krb5kdc[20854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311382, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:43:02.810783 20867 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 04:43:02.811842 20867 webserver.cc:460] Webserver started at http://127.0.0.1:46665/ using document root <none> and password file <none>
I20260925 04:43:02.812458 20867 fs_manager.cc:359] Metadata directory not provided
I20260925 04:43:02.812533 20867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:43:02.812763 20867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:43:02.814960 20867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790311123478858-392-0/minicluster-data/master-0/data/instance:
uuid: "7e985599f6f74d7e8f5f65d5969fd99b"
format_stamp: "Formatted at 2026-09-25 04:43:02 on dist-test-slave-dc95"
server_key: "9d7b46cfffe2749f0f95e0ff6e02b478"
server_key_iv: "b4b3cc53820c29191fbd3e61957f9ffd"
server_key_version: "encryptionkey@0"
I20260925 04:43:02.815559 20867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790311123478858-392-0/minicluster-data/master-0/wal/instance:
uuid: "7e985599f6f74d7e8f5f65d5969fd99b"
format_stamp: "Formatted at 2026-09-25 04:43:02 on dist-test-slave-dc95"
server_key: "9d7b46cfffe2749f0f95e0ff6e02b478"
server_key_iv: "b4b3cc53820c29191fbd3e61957f9ffd"
server_key_version: "encryptionkey@0"
I20260925 04:43:02.819695 20867 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260925 04:43:02.822489 20882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:43:02.823976 20867 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 04:43:02.824115 20867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790311123478858-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790311123478858-392-0/minicluster-data/master-0/wal
uuid: "7e985599f6f74d7e8f5f65d5969fd99b"
format_stamp: "Formatted at 2026-09-25 04:43:02 on dist-test-slave-dc95"
server_key: "9d7b46cfffe2749f0f95e0ff6e02b478"
server_key_iv: "b4b3cc53820c29191fbd3e61957f9ffd"
server_key_version: "encryptionkey@0"
I20260925 04:43:02.824234 20867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790311123478858-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790311123478858-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790311123478858-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:43:02.829098 20867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:43:02.832206 20867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:43:02.832379 20867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:43:02.839340 20867 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38479
I20260925 04:43:02.839356 20934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38479 every 8 connection(s)
I20260925 04:43:02.840353 20867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
I20260925 04:43:02.843200 20935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:43:02.844182 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 20867
I20260925 04:43:02.844331 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790311123478858-392-0/minicluster-data/master-0/wal/instance
I20260925 04:43:02.844640 392 external_mini_cluster.cc:1428] Setting key b7516ce5d5c85eb525bfcad544289e52
I20260925 04:43:02.849647 20935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e985599f6f74d7e8f5f65d5969fd99b: Bootstrap starting.
I20260925 04:43:02.849877 20938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:02.845964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59805 (local address 127.0.0.1:38479)
0925 04:43:02.846367 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:02.846386 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:02.846421 (+ 35us) server_negotiation.cc:407] Connection header received
0925 04:43:02.847055 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:02.847074 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:02.847369 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:02.847716 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:02.848574 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:43:02.848579 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:43:02.848590 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:43:02.848602 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:43:02.848663 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:43:02.848684 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:43:02.848796 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:02.848910 (+ 114us) server_negotiation.cc:299] Negotiation successful
0925 04:43:02.849056 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":98,"threads_started":1}
I20260925 04:43:02.852006 20935 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e985599f6f74d7e8f5f65d5969fd99b: Neither blocks nor log segments found. Creating new log.
I20260925 04:43:02.852571 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:43:02.852797 20935 log.cc:824] T 00000000000000000000000000000000 P 7e985599f6f74d7e8f5f65d5969fd99b: Log is configured to *not* fsync() on all Append() calls
I20260925 04:43:02.855007 20935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e985599f6f74d7e8f5f65d5969fd99b: No bootstrap required, opened a new log
I20260925 04:43:02.856143 20938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:02.854584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60569 (local address 127.0.0.1:38479)
0925 04:43:02.854690 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:02.854695 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:02.854861 (+ 166us) server_negotiation.cc:407] Connection header received
0925 04:43:02.854968 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:02.854971 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:02.855005 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:02.855101 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:02.855771 (+ 670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:43:02.855774 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:43:02.855776 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:43:02.855786 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:43:02.855810 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:43:02.855820 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:43:02.855884 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:02.855984 (+ 100us) server_negotiation.cc:299] Negotiation successful
0925 04:43:02.856014 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 04:43:02.858958 20935 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e985599f6f74d7e8f5f65d5969fd99b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e985599f6f74d7e8f5f65d5969fd99b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38479 } }
I20260925 04:43:02.859187 20935 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e985599f6f74d7e8f5f65d5969fd99b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:43:02.859220 20935 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e985599f6f74d7e8f5f65d5969fd99b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e985599f6f74d7e8f5f65d5969fd99b, State: Initialized, Role: FOLLOWER
I20260925 04:43:02.859674 20935 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e985599f6f74d7e8f5f65d5969fd99b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e985599f6f74d7e8f5f65d5969fd99b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38479 } }
I20260925 04:43:02.859805 20935 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e985599f6f74d7e8f5f65d5969fd99b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:43:02.859851 20935 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e985599f6f74d7e8f5f65d5969fd99b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:43:02.859925 20935 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e985599f6f74d7e8f5f65d5969fd99b [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:43:02.860930 20935 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e985599f6f74d7e8f5f65d5969fd99b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e985599f6f74d7e8f5f65d5969fd99b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38479 } }
I20260925 04:43:02.861268 20935 leader_election.cc:302] T 00000000000000000000000000000000 P 7e985599f6f74d7e8f5f65d5969fd99b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e985599f6f74d7e8f5f65d5969fd99b; no voters:
I20260925 04:43:02.861564 20935 leader_election.cc:288] T 00000000000000000000000000000000 P 7e985599f6f74d7e8f5f65d5969fd99b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:43:02.861665 20945 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e985599f6f74d7e8f5f65d5969fd99b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:43:02.861871 20945 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e985599f6f74d7e8f5f65d5969fd99b [term 1 LEADER]: Becoming Leader. State: Replica: 7e985599f6f74d7e8f5f65d5969fd99b, State: Running, Role: LEADER
I20260925 04:43:02.862229 20945 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e985599f6f74d7e8f5f65d5969fd99b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e985599f6f74d7e8f5f65d5969fd99b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38479 } }
I20260925 04:43:02.862718 20935 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e985599f6f74d7e8f5f65d5969fd99b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:43:02.863848 20946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e985599f6f74d7e8f5f65d5969fd99b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e985599f6f74d7e8f5f65d5969fd99b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e985599f6f74d7e8f5f65d5969fd99b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38479 } } }
I20260925 04:43:02.863832 20947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e985599f6f74d7e8f5f65d5969fd99b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e985599f6f74d7e8f5f65d5969fd99b. Latest consensus state: current_term: 1 leader_uuid: "7e985599f6f74d7e8f5f65d5969fd99b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e985599f6f74d7e8f5f65d5969fd99b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38479 } } }
I20260925 04:43:02.864007 20947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e985599f6f74d7e8f5f65d5969fd99b [sys.catalog]: This master's current role is: LEADER
I20260925 04:43:02.864239 20946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e985599f6f74d7e8f5f65d5969fd99b [sys.catalog]: This master's current role is: LEADER
I20260925 04:43:02.864604 20954 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:43:02.867377 20954 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:43:02.872567 20954 catalog_manager.cc:1347] Generated new cluster ID: 52b7c61b8c0d47f3ab6fd9a44200a84b
I20260925 04:43:02.872651 20954 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:43:02.896256 20954 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:43:02.896631 20954 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:43:02.900451 20954 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e985599f6f74d7e8f5f65d5969fd99b: Generated new TSK 0
I20260925 04:43:02.901311 20954 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:43:02.909814 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 20867
2026-09-25T04:43:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1330 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790311123478858-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:43:02 dist-test-slave-dc95 krb5kdc[20966](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:43:02 dist-test-slave-dc95 krb5kdc[20966](info): set up 2 sockets
Sep 25 04:43:02 dist-test-slave-dc95 krb5kdc[20966](info): commencing operation
krb5kdc: starting...
W20260925 04:43:03.871562 392 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790311123478858-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790311123478858-392-0/krb5kdc/test-user.keytab.
Sep 25 04:43:03 dist-test-slave-dc95 krb5kdc[20966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:43:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:43:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790311123478858-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790311123478858-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 04:43:03.999101 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790311123478858-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790311123478858-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790311123478858-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36135
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790311123478858-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36135
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:43:04.109863 20979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:43:04.110165 20979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:43:04.110208 20979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:43:04.114346 20979 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:43:04.114434 20979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:43:04.114454 20979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:43:04.114470 20979 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:43:04.114485 20979 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:43:04.120076 20979 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790311123478858-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790311123478858-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36135
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790311123478858-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790311123478858-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:43:04.121176 20979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:43:04.122182 20979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:43:04.130995 20985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:43:04.131033 20984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:43:04.131064 20979 server_base.cc:1034] running on GCE node
W20260925 04:43:04.130995 20987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:43:04.132115 20979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:43:04.133368 20979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:43:04.134573 20979 hybrid_clock.cc:648] HybridClock initialized: now 1790311384134562 us; error 31 us; skew 500 ppm
Sep 25 04:43:04 dist-test-slave-dc95 krb5kdc[20966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311384, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:43:04.142231 20979 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 04:43:04.143265 20979 webserver.cc:460] Webserver started at http://127.0.0.1:38719/ using document root <none> and password file <none>
I20260925 04:43:04.143872 20979 fs_manager.cc:359] Metadata directory not provided
I20260925 04:43:04.143942 20979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:43:04.144120 20979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:43:04.146234 20979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790311123478858-392-0/minicluster-data/master-0/data/instance:
uuid: "c6f0a13652454f18a8a8a417df2d0999"
format_stamp: "Formatted at 2026-09-25 04:43:04 on dist-test-slave-dc95"
server_key: "2857cf6dfbba47118e9eb69bb9cfe7d1"
server_key_iv: "a9ea214b70ce31d76eacc6940d33bf96"
server_key_version: "encryptionkey@0"
I20260925 04:43:04.146785 20979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790311123478858-392-0/minicluster-data/master-0/wal/instance:
uuid: "c6f0a13652454f18a8a8a417df2d0999"
format_stamp: "Formatted at 2026-09-25 04:43:04 on dist-test-slave-dc95"
server_key: "2857cf6dfbba47118e9eb69bb9cfe7d1"
server_key_iv: "a9ea214b70ce31d76eacc6940d33bf96"
server_key_version: "encryptionkey@0"
I20260925 04:43:04.150833 20979 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 04:43:04.153729 20994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:43:04.155232 20979 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 04:43:04.155380 20979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790311123478858-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790311123478858-392-0/minicluster-data/master-0/wal
uuid: "c6f0a13652454f18a8a8a417df2d0999"
format_stamp: "Formatted at 2026-09-25 04:43:04 on dist-test-slave-dc95"
server_key: "2857cf6dfbba47118e9eb69bb9cfe7d1"
server_key_iv: "a9ea214b70ce31d76eacc6940d33bf96"
server_key_version: "encryptionkey@0"
I20260925 04:43:04.155498 20979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790311123478858-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790311123478858-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790311123478858-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:43:04.175299 20979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:43:04.178692 20979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:43:04.178865 20979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:43:04.185983 20979 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36135
I20260925 04:43:04.186000 21046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36135 every 8 connection(s)
I20260925 04:43:04.187013 20979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
I20260925 04:43:04.190105 21047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:43:04.195930 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 20979
I20260925 04:43:04.196054 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790311123478858-392-0/minicluster-data/master-0/wal/instance
I20260925 04:43:04.196097 21047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6f0a13652454f18a8a8a417df2d0999: Bootstrap starting.
I20260925 04:43:04.196303 392 external_mini_cluster.cc:1428] Setting key 027de547d1906d3ba4b49cb193e5cdfb
I20260925 04:43:04.198846 21047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6f0a13652454f18a8a8a417df2d0999: Neither blocks nor log segments found. Creating new log.
I20260925 04:43:04.199658 21047 log.cc:824] T 00000000000000000000000000000000 P c6f0a13652454f18a8a8a417df2d0999: Log is configured to *not* fsync() on all Append() calls
I20260925 04:43:04.201736 21047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6f0a13652454f18a8a8a417df2d0999: No bootstrap required, opened a new log
Sep 25 04:43:04 dist-test-slave-dc95 krb5kdc[20966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 04:43:04.205307 21047 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6f0a13652454f18a8a8a417df2d0999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6f0a13652454f18a8a8a417df2d0999" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36135 } }
I20260925 04:43:04.205544 21047 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6f0a13652454f18a8a8a417df2d0999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:43:04.205646 21047 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6f0a13652454f18a8a8a417df2d0999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6f0a13652454f18a8a8a417df2d0999, State: Initialized, Role: FOLLOWER
I20260925 04:43:04.206116 21047 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6f0a13652454f18a8a8a417df2d0999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6f0a13652454f18a8a8a417df2d0999" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36135 } }
I20260925 04:43:04.206254 21047 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6f0a13652454f18a8a8a417df2d0999 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:43:04.206321 21047 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6f0a13652454f18a8a8a417df2d0999 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:43:04.206419 21047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6f0a13652454f18a8a8a417df2d0999 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:43:04.207321 21047 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6f0a13652454f18a8a8a417df2d0999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6f0a13652454f18a8a8a417df2d0999" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36135 } }
I20260925 04:43:04.207744 21047 leader_election.cc:302] T 00000000000000000000000000000000 P c6f0a13652454f18a8a8a417df2d0999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6f0a13652454f18a8a8a417df2d0999; no voters:
I20260925 04:43:04.208135 21047 leader_election.cc:288] T 00000000000000000000000000000000 P c6f0a13652454f18a8a8a417df2d0999 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:43:04.208206 21052 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6f0a13652454f18a8a8a417df2d0999 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:43:04.208413 21052 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6f0a13652454f18a8a8a417df2d0999 [term 1 LEADER]: Becoming Leader. State: Replica: c6f0a13652454f18a8a8a417df2d0999, State: Running, Role: LEADER
I20260925 04:43:04.208765 21052 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6f0a13652454f18a8a8a417df2d0999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6f0a13652454f18a8a8a417df2d0999" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36135 } }
I20260925 04:43:04.209517 21047 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6f0a13652454f18a8a8a417df2d0999 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:43:04.210448 21053 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6f0a13652454f18a8a8a417df2d0999 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6f0a13652454f18a8a8a417df2d0999. Latest consensus state: current_term: 1 leader_uuid: "c6f0a13652454f18a8a8a417df2d0999" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6f0a13652454f18a8a8a417df2d0999" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36135 } } }
I20260925 04:43:04.210620 21053 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6f0a13652454f18a8a8a417df2d0999 [sys.catalog]: This master's current role is: LEADER
I20260925 04:43:04.210937 21054 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6f0a13652454f18a8a8a417df2d0999 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6f0a13652454f18a8a8a417df2d0999" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6f0a13652454f18a8a8a417df2d0999" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36135 } } }
I20260925 04:43:04.211022 21054 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6f0a13652454f18a8a8a417df2d0999 [sys.catalog]: This master's current role is: LEADER
I20260925 04:43:04.211148 21050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:04.197781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38097 (local address 127.0.0.1:36135)
0925 04:43:04.198280 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:04.198296 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:04.198334 (+ 38us) server_negotiation.cc:407] Connection header received
0925 04:43:04.199122 (+ 788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:04.199144 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:04.199571 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:04.199953 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:04.200719 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:04.201739 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:04.202611 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:04.203098 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:04.203241 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:04.205661 (+ 2420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:43:04.205666 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:43:04.205680 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:43:04.205710 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:43:04.207399 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:04.208001 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:04.208012 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:04.208019 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:04.208152 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:04.208500 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:04.208503 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:04.208505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:04.209095 (+ 590us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:43:04.209308 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:04.209580 (+ 272us) server_negotiation.cc:299] Negotiation successful
0925 04:43:04.210059 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":153,"threads_started":1}
I20260925 04:43:04.212067 21061 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:43:04.214205 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:43:04.215510 21061 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:43:04.221812 21061 catalog_manager.cc:1347] Generated new cluster ID: 136a19ec48ec4c87abbc799f44bb8c99
I20260925 04:43:04.222157 21061 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:43:04.224644 21050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:04.216126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42511 (local address 127.0.0.1:36135)
0925 04:43:04.216258 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:04.216264 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:04.216444 (+ 180us) server_negotiation.cc:407] Connection header received
0925 04:43:04.216553 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:04.216556 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:04.216600 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:04.216664 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:04.217306 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:04.217900 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:04.219067 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:04.219400 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:04.219504 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:04.221230 (+ 1726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:43:04.221233 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:43:04.221237 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:43:04.221262 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:43:04.222924 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:04.223435 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:04.223439 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:04.223440 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:04.223525 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:04.223827 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:04.223829 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:04.223831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:04.224030 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:43:04.224140 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:04.224375 (+ 235us) server_negotiation.cc:299] Negotiation successful
0925 04:43:04.224491 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 04:43:04.240583 21061 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:43:04.241921 21061 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:43:04.251569 21061 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6f0a13652454f18a8a8a417df2d0999: Generated new TSK 0
I20260925 04:43:04.252525 21061 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:43:04.261194 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 20979
2026-09-25T04:43:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1351 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790311123478858-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:43:04 dist-test-slave-dc95 krb5kdc[21078](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:43:04 dist-test-slave-dc95 krb5kdc[21078](info): set up 2 sockets
Sep 25 04:43:04 dist-test-slave-dc95 krb5kdc[21078](info): commencing operation
krb5kdc: starting...
W20260925 04:43:05.215710 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790311123478858-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790311123478858-392-0/krb5kdc/test-user.keytab.
Sep 25 04:43:05 dist-test-slave-dc95 krb5kdc[21078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:43:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:43:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790311123478858-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790311123478858-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 04:43:05.339562 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790311123478858-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790311123478858-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790311123478858-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45863
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790311123478858-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45863
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:43:05.451330 21091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:43:05.451697 21091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:43:05.451774 21091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:43:05.456000 21091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:43:05.456076 21091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:43:05.456095 21091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:43:05.456111 21091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:43:05.456126 21091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:43:05.461815 21091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790311123478858-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790311123478858-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790311123478858-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790311123478858-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:43:05.462973 21091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:43:05.464023 21091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:43:05.472579 21097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:43:05.472558 21099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:43:05.472558 21096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:43:05.472646 21091 server_base.cc:1034] running on GCE node
I20260925 04:43:05.473410 21091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:43:05.474577 21091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:43:05.475741 21091 hybrid_clock.cc:648] HybridClock initialized: now 1790311385475722 us; error 38 us; skew 500 ppm
Sep 25 04:43:05 dist-test-slave-dc95 krb5kdc[21078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311385, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:43:05.478736 21091 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 04:43:05.479815 21091 webserver.cc:460] Webserver started at http://127.0.0.1:44331/ using document root <none> and password file <none>
I20260925 04:43:05.480394 21091 fs_manager.cc:359] Metadata directory not provided
I20260925 04:43:05.480463 21091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:43:05.480684 21091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:43:05.482781 21091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790311123478858-392-0/minicluster-data/master-0/data/instance:
uuid: "03c08fa33a634829869d1e3a9095ada9"
format_stamp: "Formatted at 2026-09-25 04:43:05 on dist-test-slave-dc95"
server_key: "a7eb5f602d0e5251c17a3a695c7ac1ae"
server_key_iv: "c1c47b418a3c865dcbb2395335fe6e53"
server_key_version: "encryptionkey@0"
I20260925 04:43:05.483353 21091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790311123478858-392-0/minicluster-data/master-0/wal/instance:
uuid: "03c08fa33a634829869d1e3a9095ada9"
format_stamp: "Formatted at 2026-09-25 04:43:05 on dist-test-slave-dc95"
server_key: "a7eb5f602d0e5251c17a3a695c7ac1ae"
server_key_iv: "c1c47b418a3c865dcbb2395335fe6e53"
server_key_version: "encryptionkey@0"
I20260925 04:43:05.487443 21091 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260925 04:43:05.490128 21106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:43:05.491559 21091 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:43:05.491767 21091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790311123478858-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790311123478858-392-0/minicluster-data/master-0/wal
uuid: "03c08fa33a634829869d1e3a9095ada9"
format_stamp: "Formatted at 2026-09-25 04:43:05 on dist-test-slave-dc95"
server_key: "a7eb5f602d0e5251c17a3a695c7ac1ae"
server_key_iv: "c1c47b418a3c865dcbb2395335fe6e53"
server_key_version: "encryptionkey@0"
I20260925 04:43:05.491894 21091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790311123478858-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790311123478858-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790311123478858-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:43:05.502321 21091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:43:05.505906 21091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:43:05.506089 21091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:43:05.513150 21091 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45863
I20260925 04:43:05.513159 21158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45863 every 8 connection(s)
I20260925 04:43:05.514086 21091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
I20260925 04:43:05.516160 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 21091
I20260925 04:43:05.516300 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790311123478858-392-0/minicluster-data/master-0/wal/instance
I20260925 04:43:05.516584 392 external_mini_cluster.cc:1428] Setting key 8dc1754a0724787beb5010437650eb84
I20260925 04:43:05.516983 21159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:43:05.524180 21159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03c08fa33a634829869d1e3a9095ada9: Bootstrap starting.
I20260925 04:43:05.526585 21162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:05.517971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57811 (local address 127.0.0.1:45863)
0925 04:43:05.518403 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:05.518422 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:05.518463 (+ 41us) server_negotiation.cc:407] Connection header received
0925 04:43:05.519205 (+ 742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:05.519228 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:05.519717 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:05.520106 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:05.520536 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:05.521976 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:05.522895 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:05.523599 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:05.523792 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:05.524579 (+ 787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:43:05.524585 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:43:05.524601 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:43:05.525086 (+ 485us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:43:05.525153 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:43:05.525184 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:43:05.525291 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:05.525441 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 04:43:05.525663 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":127,"threads_started":1}
I20260925 04:43:05.526892 21159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03c08fa33a634829869d1e3a9095ada9: Neither blocks nor log segments found. Creating new log.
I20260925 04:43:05.527647 21159 log.cc:824] T 00000000000000000000000000000000 P 03c08fa33a634829869d1e3a9095ada9: Log is configured to *not* fsync() on all Append() calls
I20260925 04:43:05.529497 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:43:05.529640 21159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03c08fa33a634829869d1e3a9095ada9: No bootstrap required, opened a new log
I20260925 04:43:05.533314 21159 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03c08fa33a634829869d1e3a9095ada9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03c08fa33a634829869d1e3a9095ada9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45863 } }
I20260925 04:43:05.533563 21159 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03c08fa33a634829869d1e3a9095ada9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:43:05.533622 21159 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03c08fa33a634829869d1e3a9095ada9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03c08fa33a634829869d1e3a9095ada9, State: Initialized, Role: FOLLOWER
I20260925 04:43:05.534045 21159 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03c08fa33a634829869d1e3a9095ada9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03c08fa33a634829869d1e3a9095ada9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45863 } }
I20260925 04:43:05.534165 21159 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03c08fa33a634829869d1e3a9095ada9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:43:05.534226 21159 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03c08fa33a634829869d1e3a9095ada9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:43:05.534319 21159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03c08fa33a634829869d1e3a9095ada9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:43:05.535182 21162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:05.531414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45961 (local address 127.0.0.1:45863)
0925 04:43:05.531538 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:05.531543 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:05.531772 (+ 229us) server_negotiation.cc:407] Connection header received
0925 04:43:05.531883 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:05.531886 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:05.531927 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:05.531991 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:05.532243 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:05.532827 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:05.533600 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:05.533901 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:05.534011 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:05.534783 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:43:05.534786 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:43:05.534789 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:43:05.534802 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:43:05.534829 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:43:05.534839 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:43:05.534897 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:05.534981 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 04:43:05.535035 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 04:43:05.535220 21159 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03c08fa33a634829869d1e3a9095ada9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03c08fa33a634829869d1e3a9095ada9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45863 } }
I20260925 04:43:05.535574 21159 leader_election.cc:302] T 00000000000000000000000000000000 P 03c08fa33a634829869d1e3a9095ada9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03c08fa33a634829869d1e3a9095ada9; no voters:
I20260925 04:43:05.535921 21159 leader_election.cc:288] T 00000000000000000000000000000000 P 03c08fa33a634829869d1e3a9095ada9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:43:05.536010 21169 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03c08fa33a634829869d1e3a9095ada9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:43:05.536162 21169 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03c08fa33a634829869d1e3a9095ada9 [term 1 LEADER]: Becoming Leader. State: Replica: 03c08fa33a634829869d1e3a9095ada9, State: Running, Role: LEADER
I20260925 04:43:05.536461 21169 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03c08fa33a634829869d1e3a9095ada9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03c08fa33a634829869d1e3a9095ada9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45863 } }
I20260925 04:43:05.537246 21159 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03c08fa33a634829869d1e3a9095ada9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:43:05.537675 21169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03c08fa33a634829869d1e3a9095ada9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "03c08fa33a634829869d1e3a9095ada9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03c08fa33a634829869d1e3a9095ada9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45863 } } }
I20260925 04:43:05.537792 21169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03c08fa33a634829869d1e3a9095ada9 [sys.catalog]: This master's current role is: LEADER
I20260925 04:43:05.537976 21169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03c08fa33a634829869d1e3a9095ada9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03c08fa33a634829869d1e3a9095ada9. Latest consensus state: current_term: 1 leader_uuid: "03c08fa33a634829869d1e3a9095ada9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03c08fa33a634829869d1e3a9095ada9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45863 } } }
I20260925 04:43:05.538091 21169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03c08fa33a634829869d1e3a9095ada9 [sys.catalog]: This master's current role is: LEADER
I20260925 04:43:05.538409 21173 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:43:05.541715 21173 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:43:05.548022 21173 catalog_manager.cc:1347] Generated new cluster ID: 22c4ee4501e64981a82be0a5dec776ad
I20260925 04:43:05.548101 21173 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:43:05.571750 21173 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:43:05.572837 21173 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:43:05.578696 21173 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03c08fa33a634829869d1e3a9095ada9: Generated new TSK 0
I20260925 04:43:05.579511 21173 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:43:05.586733 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 21091
2026-09-25T04:43:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1325 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790311123478858-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:43:05 dist-test-slave-dc95 krb5kdc[21190](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:43:05 dist-test-slave-dc95 krb5kdc[21190](info): set up 2 sockets
Sep 25 04:43:05 dist-test-slave-dc95 krb5kdc[21190](info): commencing operation
krb5kdc: starting...
W20260925 04:43:06.547755 392 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790311123478858-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790311123478858-392-0/krb5kdc/test-user.keytab.
Sep 25 04:43:06 dist-test-slave-dc95 krb5kdc[21190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:43:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:43:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790311123478858-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790311123478858-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 04:43:06.671643 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790311123478858-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790311123478858-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790311123478858-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46257
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790311123478858-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46257
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:43:06.783203 21203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:43:06.783525 21203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:43:06.783588 21203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:43:06.788333 21203 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:43:06.788441 21203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:43:06.788470 21203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:43:06.788502 21203 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:43:06.788524 21203 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:43:06.794605 21203 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790311123478858-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790311123478858-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46257
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790311123478858-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21203
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790311123478858-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:43:06.795879 21203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:43:06.796949 21203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:43:06.805935 21208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:43:06.805989 21209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:43:06.806030 21203 server_base.cc:1034] running on GCE node
W20260925 04:43:06.805945 21211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:43:06.806931 21203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:43:06.808375 21203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:43:06.809612 21203 hybrid_clock.cc:648] HybridClock initialized: now 1790311386809583 us; error 47 us; skew 500 ppm
Sep 25 04:43:06 dist-test-slave-dc95 krb5kdc[21190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311386, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:43:06.812693 21203 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 04:43:06.813746 21203 webserver.cc:460] Webserver started at http://127.0.0.1:34541/ using document root <none> and password file <none>
I20260925 04:43:06.814325 21203 fs_manager.cc:359] Metadata directory not provided
I20260925 04:43:06.814395 21203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:43:06.814606 21203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:43:06.816871 21203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790311123478858-392-0/minicluster-data/master-0/data/instance:
uuid: "7eac5a2282564f9399978297e28421cb"
format_stamp: "Formatted at 2026-09-25 04:43:06 on dist-test-slave-dc95"
server_key: "847c3670bef4a84412b3e0e1e6e4b76b"
server_key_iv: "ab805819500547b56605dcf9cd300e94"
server_key_version: "encryptionkey@0"
I20260925 04:43:06.817494 21203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790311123478858-392-0/minicluster-data/master-0/wal/instance:
uuid: "7eac5a2282564f9399978297e28421cb"
format_stamp: "Formatted at 2026-09-25 04:43:06 on dist-test-slave-dc95"
server_key: "847c3670bef4a84412b3e0e1e6e4b76b"
server_key_iv: "ab805819500547b56605dcf9cd300e94"
server_key_version: "encryptionkey@0"
I20260925 04:43:06.821537 21203 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 04:43:06.824298 21218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:43:06.825718 21203 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:43:06.825850 21203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790311123478858-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790311123478858-392-0/minicluster-data/master-0/wal
uuid: "7eac5a2282564f9399978297e28421cb"
format_stamp: "Formatted at 2026-09-25 04:43:06 on dist-test-slave-dc95"
server_key: "847c3670bef4a84412b3e0e1e6e4b76b"
server_key_iv: "ab805819500547b56605dcf9cd300e94"
server_key_version: "encryptionkey@0"
I20260925 04:43:06.825966 21203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790311123478858-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790311123478858-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790311123478858-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:43:06.830850 21203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:43:06.834046 21203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:43:06.834232 21203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:43:06.841259 21203 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46257
I20260925 04:43:06.841286 21270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46257 every 8 connection(s)
I20260925 04:43:06.842226 21203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
I20260925 04:43:06.845151 21271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:43:06.848064 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 21203
I20260925 04:43:06.848209 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790311123478858-392-0/minicluster-data/master-0/wal/instance
I20260925 04:43:06.848485 392 external_mini_cluster.cc:1428] Setting key ae561c5a94de826e3899cacbccce9d41
I20260925 04:43:06.851464 21271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eac5a2282564f9399978297e28421cb: Bootstrap starting.
I20260925 04:43:06.853678 21274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:06.849923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49361 (local address 127.0.0.1:46257)
0925 04:43:06.850378 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:06.850393 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:06.850429 (+ 36us) server_negotiation.cc:407] Connection header received
0925 04:43:06.851073 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:06.851091 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:06.851394 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:06.851773 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:06.852499 (+ 726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:43:06.852503 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:43:06.852515 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:43:06.852525 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:43:06.852585 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:43:06.852604 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:43:06.852725 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:06.852795 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 04:43:06.852951 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":127,"threads_started":1}
I20260925 04:43:06.853931 21271 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7eac5a2282564f9399978297e28421cb: Neither blocks nor log segments found. Creating new log.
I20260925 04:43:06.854688 21271 log.cc:824] T 00000000000000000000000000000000 P 7eac5a2282564f9399978297e28421cb: Log is configured to *not* fsync() on all Append() calls
I20260925 04:43:06.856542 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:43:06.856878 21271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eac5a2282564f9399978297e28421cb: No bootstrap required, opened a new log
I20260925 04:43:06.860213 21274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:06.858535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45131 (local address 127.0.0.1:46257)
0925 04:43:06.858639 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:06.858642 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:06.858868 (+ 226us) server_negotiation.cc:407] Connection header received
0925 04:43:06.858985 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:06.858988 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:06.859023 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:06.859095 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:06.859835 (+ 740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:43:06.859837 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:43:06.859839 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:43:06.859849 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:43:06.859875 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:43:06.859885 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:43:06.859958 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:06.860033 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 04:43:06.860078 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 04:43:06.861049 21271 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7eac5a2282564f9399978297e28421cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eac5a2282564f9399978297e28421cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46257 } }
I20260925 04:43:06.861338 21271 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7eac5a2282564f9399978297e28421cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:43:06.861382 21271 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7eac5a2282564f9399978297e28421cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eac5a2282564f9399978297e28421cb, State: Initialized, Role: FOLLOWER
I20260925 04:43:06.861856 21271 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7eac5a2282564f9399978297e28421cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eac5a2282564f9399978297e28421cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46257 } }
I20260925 04:43:06.862022 21271 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7eac5a2282564f9399978297e28421cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:43:06.862092 21271 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7eac5a2282564f9399978297e28421cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:43:06.862193 21271 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7eac5a2282564f9399978297e28421cb [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:43:06.863292 21271 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7eac5a2282564f9399978297e28421cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eac5a2282564f9399978297e28421cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46257 } }
I20260925 04:43:06.863679 21271 leader_election.cc:302] T 00000000000000000000000000000000 P 7eac5a2282564f9399978297e28421cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7eac5a2282564f9399978297e28421cb; no voters:
I20260925 04:43:06.864018 21271 leader_election.cc:288] T 00000000000000000000000000000000 P 7eac5a2282564f9399978297e28421cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:43:06.864140 21281 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7eac5a2282564f9399978297e28421cb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:43:06.864430 21281 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7eac5a2282564f9399978297e28421cb [term 1 LEADER]: Becoming Leader. State: Replica: 7eac5a2282564f9399978297e28421cb, State: Running, Role: LEADER
I20260925 04:43:06.864732 21281 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7eac5a2282564f9399978297e28421cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eac5a2282564f9399978297e28421cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46257 } }
I20260925 04:43:06.865231 21271 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7eac5a2282564f9399978297e28421cb [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:43:06.866405 21283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eac5a2282564f9399978297e28421cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7eac5a2282564f9399978297e28421cb. Latest consensus state: current_term: 1 leader_uuid: "7eac5a2282564f9399978297e28421cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eac5a2282564f9399978297e28421cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46257 } } }
I20260925 04:43:06.866514 21283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eac5a2282564f9399978297e28421cb [sys.catalog]: This master's current role is: LEADER
I20260925 04:43:06.866911 21290 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:43:06.866849 21282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eac5a2282564f9399978297e28421cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7eac5a2282564f9399978297e28421cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eac5a2282564f9399978297e28421cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46257 } } }
I20260925 04:43:06.867000 21282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eac5a2282564f9399978297e28421cb [sys.catalog]: This master's current role is: LEADER
I20260925 04:43:06.869683 21290 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:43:06.875975 21290 catalog_manager.cc:1347] Generated new cluster ID: ceb07da8006f4e5b8d0478ac1480dbd8
I20260925 04:43:06.876070 21290 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:43:06.882661 21290 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:43:06.883003 21290 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:43:06.891086 21290 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7eac5a2282564f9399978297e28421cb: Generated new TSK 0
I20260925 04:43:06.891902 21290 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:43:06.900131 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 21203
2026-09-25T04:43:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1312 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790311123478858-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:43:06 dist-test-slave-dc95 krb5kdc[21302](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:43:06 dist-test-slave-dc95 krb5kdc[21302](info): set up 2 sockets
Sep 25 04:43:06 dist-test-slave-dc95 krb5kdc[21302](info): commencing operation
krb5kdc: starting...
W20260925 04:43:07.855581 392 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790311123478858-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790311123478858-392-0/krb5kdc/test-user.keytab.
Sep 25 04:43:07 dist-test-slave-dc95 krb5kdc[21302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:43:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:43:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790311123478858-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790311123478858-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 04:43:07.981030 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790311123478858-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790311123478858-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790311123478858-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35613
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790311123478858-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35613
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:43:08.093371 21315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:43:08.093683 21315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:43:08.093727 21315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:43:08.097939 21315 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:43:08.098026 21315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:43:08.098055 21315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:43:08.098078 21315 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:43:08.098150 21315 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:43:08.103894 21315 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790311123478858-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790311123478858-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35613
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790311123478858-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790311123478858-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:43:08.105038 21315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:43:08.106092 21315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:43:08.114882 21321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:43:08.114841 21320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:43:08.115064 21315 server_base.cc:1034] running on GCE node
W20260925 04:43:08.115245 21323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:43:08.115881 21315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:43:08.117086 21315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:43:08.118279 21315 hybrid_clock.cc:648] HybridClock initialized: now 1790311388118234 us; error 62 us; skew 500 ppm
Sep 25 04:43:08 dist-test-slave-dc95 krb5kdc[21302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:43:08.121230 21315 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 04:43:08.122154 21315 webserver.cc:460] Webserver started at http://127.0.0.1:34447/ using document root <none> and password file <none>
I20260925 04:43:08.122704 21315 fs_manager.cc:359] Metadata directory not provided
I20260925 04:43:08.122769 21315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:43:08.122969 21315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:43:08.125039 21315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790311123478858-392-0/minicluster-data/master-0/data/instance:
uuid: "e288f93bdd6e42de98c6c4c4ac172d6d"
format_stamp: "Formatted at 2026-09-25 04:43:08 on dist-test-slave-dc95"
server_key: "9d21e7051d819c02a5b91fb049d7c3e1"
server_key_iv: "c7547306bc02e44abc702cdf2c861fe1"
server_key_version: "encryptionkey@0"
I20260925 04:43:08.125590 21315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790311123478858-392-0/minicluster-data/master-0/wal/instance:
uuid: "e288f93bdd6e42de98c6c4c4ac172d6d"
format_stamp: "Formatted at 2026-09-25 04:43:08 on dist-test-slave-dc95"
server_key: "9d21e7051d819c02a5b91fb049d7c3e1"
server_key_iv: "c7547306bc02e44abc702cdf2c861fe1"
server_key_version: "encryptionkey@0"
I20260925 04:43:08.129575 21315 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260925 04:43:08.132467 21330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:43:08.133853 21315 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 04:43:08.134030 21315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790311123478858-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790311123478858-392-0/minicluster-data/master-0/wal
uuid: "e288f93bdd6e42de98c6c4c4ac172d6d"
format_stamp: "Formatted at 2026-09-25 04:43:08 on dist-test-slave-dc95"
server_key: "9d21e7051d819c02a5b91fb049d7c3e1"
server_key_iv: "c7547306bc02e44abc702cdf2c861fe1"
server_key_version: "encryptionkey@0"
I20260925 04:43:08.134141 21315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790311123478858-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790311123478858-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790311123478858-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:43:08.160753 21315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:43:08.164247 21315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:43:08.164455 21315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:43:08.171552 21315 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35613
I20260925 04:43:08.171649 21382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35613 every 8 connection(s)
I20260925 04:43:08.172596 21315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
I20260925 04:43:08.175568 21383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:43:08.178014 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 21315
I20260925 04:43:08.178151 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790311123478858-392-0/minicluster-data/master-0/wal/instance
I20260925 04:43:08.178428 392 external_mini_cluster.cc:1428] Setting key b70bcd2f37abb6288f93359a63fde9cb
I20260925 04:43:08.181869 21383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e288f93bdd6e42de98c6c4c4ac172d6d: Bootstrap starting.
I20260925 04:43:08.184197 21383 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e288f93bdd6e42de98c6c4c4ac172d6d: Neither blocks nor log segments found. Creating new log.
I20260925 04:43:08.184928 21383 log.cc:824] T 00000000000000000000000000000000 P e288f93bdd6e42de98c6c4c4ac172d6d: Log is configured to *not* fsync() on all Append() calls
I20260925 04:43:08.186960 21383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e288f93bdd6e42de98c6c4c4ac172d6d: No bootstrap required, opened a new log
Sep 25 04:43:08 dist-test-slave-dc95 krb5kdc[21302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 04:43:08.190340 21383 raft_consensus.cc:348] T 00000000000000000000000000000000 P e288f93bdd6e42de98c6c4c4ac172d6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e288f93bdd6e42de98c6c4c4ac172d6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35613 } }
I20260925 04:43:08.190580 21383 raft_consensus.cc:374] T 00000000000000000000000000000000 P e288f93bdd6e42de98c6c4c4ac172d6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:43:08.190634 21383 raft_consensus.cc:729] T 00000000000000000000000000000000 P e288f93bdd6e42de98c6c4c4ac172d6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e288f93bdd6e42de98c6c4c4ac172d6d, State: Initialized, Role: FOLLOWER
I20260925 04:43:08.191082 21383 consensus_queue.cc:260] T 00000000000000000000000000000000 P e288f93bdd6e42de98c6c4c4ac172d6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e288f93bdd6e42de98c6c4c4ac172d6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35613 } }
I20260925 04:43:08.191218 21383 raft_consensus.cc:388] T 00000000000000000000000000000000 P e288f93bdd6e42de98c6c4c4ac172d6d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:43:08.191282 21383 raft_consensus.cc:482] T 00000000000000000000000000000000 P e288f93bdd6e42de98c6c4c4ac172d6d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:43:08.191385 21383 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e288f93bdd6e42de98c6c4c4ac172d6d [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:43:08.192308 21383 raft_consensus.cc:504] T 00000000000000000000000000000000 P e288f93bdd6e42de98c6c4c4ac172d6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e288f93bdd6e42de98c6c4c4ac172d6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35613 } }
I20260925 04:43:08.192639 21383 leader_election.cc:302] T 00000000000000000000000000000000 P e288f93bdd6e42de98c6c4c4ac172d6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e288f93bdd6e42de98c6c4c4ac172d6d; no voters:
I20260925 04:43:08.192937 21383 leader_election.cc:288] T 00000000000000000000000000000000 P e288f93bdd6e42de98c6c4c4ac172d6d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:43:08.193029 21388 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e288f93bdd6e42de98c6c4c4ac172d6d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:43:08.193230 21388 raft_consensus.cc:686] T 00000000000000000000000000000000 P e288f93bdd6e42de98c6c4c4ac172d6d [term 1 LEADER]: Becoming Leader. State: Replica: e288f93bdd6e42de98c6c4c4ac172d6d, State: Running, Role: LEADER
I20260925 04:43:08.193600 21388 consensus_queue.cc:237] T 00000000000000000000000000000000 P e288f93bdd6e42de98c6c4c4ac172d6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e288f93bdd6e42de98c6c4c4ac172d6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35613 } }
I20260925 04:43:08.193727 21386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:08.179883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:53027 (local address 127.0.0.1:35613)
0925 04:43:08.180375 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:08.180391 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:08.180443 (+ 52us) server_negotiation.cc:407] Connection header received
0925 04:43:08.181142 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:08.181162 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:08.181548 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:08.181893 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:08.182719 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:08.183787 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:08.184668 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:08.185169 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:08.185310 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:08.188143 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:43:08.188148 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:43:08.188162 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:43:08.188190 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:43:08.190427 (+ 2237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:08.190997 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:08.191002 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:08.191007 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:08.191119 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:08.191439 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:08.191442 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:08.191444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:08.191906 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:43:08.192111 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:08.192431 (+ 320us) server_negotiation.cc:299] Negotiation successful
0925 04:43:08.192673 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":140,"threads_started":1}
I20260925 04:43:08.194274 21383 sys_catalog.cc:564] T 00000000000000000000000000000000 P e288f93bdd6e42de98c6c4c4ac172d6d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:43:08.195339 21389 sys_catalog.cc:455] T 00000000000000000000000000000000 P e288f93bdd6e42de98c6c4c4ac172d6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e288f93bdd6e42de98c6c4c4ac172d6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e288f93bdd6e42de98c6c4c4ac172d6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35613 } } }
I20260925 04:43:08.195480 21389 sys_catalog.cc:458] T 00000000000000000000000000000000 P e288f93bdd6e42de98c6c4c4ac172d6d [sys.catalog]: This master's current role is: LEADER
I20260925 04:43:08.195815 21390 sys_catalog.cc:455] T 00000000000000000000000000000000 P e288f93bdd6e42de98c6c4c4ac172d6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e288f93bdd6e42de98c6c4c4ac172d6d. Latest consensus state: current_term: 1 leader_uuid: "e288f93bdd6e42de98c6c4c4ac172d6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e288f93bdd6e42de98c6c4c4ac172d6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35613 } } }
I20260925 04:43:08.195914 21390 sys_catalog.cc:458] T 00000000000000000000000000000000 P e288f93bdd6e42de98c6c4c4ac172d6d [sys.catalog]: This master's current role is: LEADER
I20260925 04:43:08.196751 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260925 04:43:08.199085 21403 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e288f93bdd6e42de98c6c4c4ac172d6d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 04:43:08.199213 21403 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 04:43:08.199291 21397 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:43:08.200340 21397 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:43:08.207650 21386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:08.199185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:59625 (local address 127.0.0.1:35613)
0925 04:43:08.199302 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:08.199306 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:08.199724 (+ 418us) server_negotiation.cc:407] Connection header received
0925 04:43:08.199889 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:08.199892 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:08.199940 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:08.200003 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:08.200671 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:08.201216 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:08.201987 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:08.202219 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:08.202306 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:08.203668 (+ 1362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:43:08.203672 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:43:08.203675 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:43:08.203701 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:43:08.205027 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:08.206018 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:08.206022 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:08.206024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:08.206119 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:08.206506 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:08.206509 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:08.206511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:08.206745 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:43:08.206901 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:08.207246 (+ 345us) server_negotiation.cc:299] Negotiation successful
0925 04:43:08.207359 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 04:43:08.207728 21397 catalog_manager.cc:1347] Generated new cluster ID: 2e19f2564cba4d0da8b93fd60191fd9e
I20260925 04:43:08.207883 21397 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:43:08.233615 21397 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:43:08.234978 21397 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:43:08.251546 21397 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e288f93bdd6e42de98c6c4c4ac172d6d: Generated new TSK 0
I20260925 04:43:08.252331 21397 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:43:08.267211 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 21315
W20260925 04:43:08.272222 21300 connection.cc:535] client connection to 127.0.0.1:35613 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35613): Connection reset by peer (error 104)
2026-09-25T04:43:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1367 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790311123478858-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:43:08 dist-test-slave-dc95 krb5kdc[21414](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:43:08 dist-test-slave-dc95 krb5kdc[21414](info): set up 2 sockets
Sep 25 04:43:08 dist-test-slave-dc95 krb5kdc[21414](info): commencing operation
krb5kdc: starting...
W20260925 04:43:09.227705 392 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790311123478858-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790311123478858-392-0/krb5kdc/test-user.keytab.
Sep 25 04:43:09 dist-test-slave-dc95 krb5kdc[21414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:43:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:43:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790311123478858-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790311123478858-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 04:43:09.354934 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790311123478858-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790311123478858-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790311123478858-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44345
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790311123478858-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44345
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:43:09.468693 21427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:43:09.469017 21427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:43:09.469079 21427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:43:09.473649 21427 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:43:09.473744 21427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:43:09.473773 21427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:43:09.473796 21427 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:43:09.473819 21427 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:43:09.479565 21427 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790311123478858-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790311123478858-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44345
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790311123478858-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790311123478858-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:43:09.480765 21427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:43:09.481849 21427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:43:09.490711 21433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:43:09.490687 21435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:43:09.490810 21427 server_base.cc:1034] running on GCE node
W20260925 04:43:09.490667 21432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:43:09.491529 21427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:43:09.492801 21427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:43:09.493992 21427 hybrid_clock.cc:648] HybridClock initialized: now 1790311389493948 us; error 63 us; skew 500 ppm
Sep 25 04:43:09 dist-test-slave-dc95 krb5kdc[21414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311389, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:43:09.502509 21427 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 04:43:09.503582 21427 webserver.cc:460] Webserver started at http://127.0.0.1:41737/ using document root <none> and password file <none>
I20260925 04:43:09.504222 21427 fs_manager.cc:359] Metadata directory not provided
I20260925 04:43:09.504290 21427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:43:09.504484 21427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:43:09.506636 21427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790311123478858-392-0/minicluster-data/master-0/data/instance:
uuid: "b37c4cee696742f69da8d6501fc6d2ac"
format_stamp: "Formatted at 2026-09-25 04:43:09 on dist-test-slave-dc95"
server_key: "423c897386af04381f560f3db4b8d0bf"
server_key_iv: "e4e594df2f6393d3b1a23fbce574a90b"
server_key_version: "encryptionkey@0"
I20260925 04:43:09.507213 21427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790311123478858-392-0/minicluster-data/master-0/wal/instance:
uuid: "b37c4cee696742f69da8d6501fc6d2ac"
format_stamp: "Formatted at 2026-09-25 04:43:09 on dist-test-slave-dc95"
server_key: "423c897386af04381f560f3db4b8d0bf"
server_key_iv: "e4e594df2f6393d3b1a23fbce574a90b"
server_key_version: "encryptionkey@0"
I20260925 04:43:09.511234 21427 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260925 04:43:09.513836 21442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:43:09.515210 21427 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:43:09.515328 21427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790311123478858-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790311123478858-392-0/minicluster-data/master-0/wal
uuid: "b37c4cee696742f69da8d6501fc6d2ac"
format_stamp: "Formatted at 2026-09-25 04:43:09 on dist-test-slave-dc95"
server_key: "423c897386af04381f560f3db4b8d0bf"
server_key_iv: "e4e594df2f6393d3b1a23fbce574a90b"
server_key_version: "encryptionkey@0"
I20260925 04:43:09.515455 21427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790311123478858-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790311123478858-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790311123478858-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:43:09.553706 21427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:43:09.557264 21427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:43:09.557446 21427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:43:09.564514 21427 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44345
I20260925 04:43:09.564523 21494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44345 every 8 connection(s)
I20260925 04:43:09.565546 21427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
I20260925 04:43:09.568442 21495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:43:09.572474 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 21427
I20260925 04:43:09.572628 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790311123478858-392-0/minicluster-data/master-0/wal/instance
I20260925 04:43:09.572899 392 external_mini_cluster.cc:1428] Setting key 6816a359ac852e12357c25179e92fa95
I20260925 04:43:09.574719 21495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b37c4cee696742f69da8d6501fc6d2ac: Bootstrap starting.
I20260925 04:43:09.577014 21495 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b37c4cee696742f69da8d6501fc6d2ac: Neither blocks nor log segments found. Creating new log.
I20260925 04:43:09.577872 21495 log.cc:824] T 00000000000000000000000000000000 P b37c4cee696742f69da8d6501fc6d2ac: Log is configured to *not* fsync() on all Append() calls
I20260925 04:43:09.580406 21495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b37c4cee696742f69da8d6501fc6d2ac: No bootstrap required, opened a new log
I20260925 04:43:09.582857 21498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:09.574435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:49349 (local address 127.0.0.1:44345)
0925 04:43:09.574912 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:09.574929 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:09.574966 (+ 37us) server_negotiation.cc:407] Connection header received
0925 04:43:09.575699 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:09.575719 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:09.576079 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:09.576398 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:09.576819 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:09.577891 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:09.578952 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:09.579454 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:09.579578 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:09.580585 (+ 1007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:43:09.580591 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:43:09.580606 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:43:09.581035 (+ 429us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:43:09.581109 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:43:09.581136 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:43:09.581255 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:09.581520 (+ 265us) server_negotiation.cc:299] Negotiation successful
0925 04:43:09.581681 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":139,"threads_started":1}
I20260925 04:43:09.583777 21495 raft_consensus.cc:348] T 00000000000000000000000000000000 P b37c4cee696742f69da8d6501fc6d2ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b37c4cee696742f69da8d6501fc6d2ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44345 } }
I20260925 04:43:09.584024 21495 raft_consensus.cc:374] T 00000000000000000000000000000000 P b37c4cee696742f69da8d6501fc6d2ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:43:09.584074 21495 raft_consensus.cc:729] T 00000000000000000000000000000000 P b37c4cee696742f69da8d6501fc6d2ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b37c4cee696742f69da8d6501fc6d2ac, State: Initialized, Role: FOLLOWER
I20260925 04:43:09.584525 21495 consensus_queue.cc:260] T 00000000000000000000000000000000 P b37c4cee696742f69da8d6501fc6d2ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b37c4cee696742f69da8d6501fc6d2ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44345 } }
I20260925 04:43:09.584664 21495 raft_consensus.cc:388] T 00000000000000000000000000000000 P b37c4cee696742f69da8d6501fc6d2ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:43:09.584734 21495 raft_consensus.cc:482] T 00000000000000000000000000000000 P b37c4cee696742f69da8d6501fc6d2ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:43:09.584831 21495 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b37c4cee696742f69da8d6501fc6d2ac [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:43:09.585999 21495 raft_consensus.cc:504] T 00000000000000000000000000000000 P b37c4cee696742f69da8d6501fc6d2ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b37c4cee696742f69da8d6501fc6d2ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44345 } }
I20260925 04:43:09.586382 21495 leader_election.cc:302] T 00000000000000000000000000000000 P b37c4cee696742f69da8d6501fc6d2ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b37c4cee696742f69da8d6501fc6d2ac; no voters:
I20260925 04:43:09.586443 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:43:09.586762 21495 leader_election.cc:288] T 00000000000000000000000000000000 P b37c4cee696742f69da8d6501fc6d2ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:43:09.586885 21500 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b37c4cee696742f69da8d6501fc6d2ac [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:43:09.587106 21500 raft_consensus.cc:686] T 00000000000000000000000000000000 P b37c4cee696742f69da8d6501fc6d2ac [term 1 LEADER]: Becoming Leader. State: Replica: b37c4cee696742f69da8d6501fc6d2ac, State: Running, Role: LEADER
I20260925 04:43:09.587512 21500 consensus_queue.cc:237] T 00000000000000000000000000000000 P b37c4cee696742f69da8d6501fc6d2ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b37c4cee696742f69da8d6501fc6d2ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44345 } }
I20260925 04:43:09.588213 21495 sys_catalog.cc:564] T 00000000000000000000000000000000 P b37c4cee696742f69da8d6501fc6d2ac [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:43:09.589148 21501 sys_catalog.cc:455] T 00000000000000000000000000000000 P b37c4cee696742f69da8d6501fc6d2ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b37c4cee696742f69da8d6501fc6d2ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b37c4cee696742f69da8d6501fc6d2ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44345 } } }
I20260925 04:43:09.589315 21501 sys_catalog.cc:458] T 00000000000000000000000000000000 P b37c4cee696742f69da8d6501fc6d2ac [sys.catalog]: This master's current role is: LEADER
I20260925 04:43:09.589617 21502 sys_catalog.cc:455] T 00000000000000000000000000000000 P b37c4cee696742f69da8d6501fc6d2ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader b37c4cee696742f69da8d6501fc6d2ac. Latest consensus state: current_term: 1 leader_uuid: "b37c4cee696742f69da8d6501fc6d2ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b37c4cee696742f69da8d6501fc6d2ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44345 } } }
I20260925 04:43:09.589702 21502 sys_catalog.cc:458] T 00000000000000000000000000000000 P b37c4cee696742f69da8d6501fc6d2ac [sys.catalog]: This master's current role is: LEADER
W20260925 04:43:09.593233 21520 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b37c4cee696742f69da8d6501fc6d2ac: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 04:43:09.593405 21520 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 04:43:09.593411 21498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:09.588705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:34731 (local address 127.0.0.1:44345)
0925 04:43:09.588812 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:09.588816 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:09.589290 (+ 474us) server_negotiation.cc:407] Connection header received
0925 04:43:09.589493 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:09.589497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:09.589552 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:09.589639 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:09.589944 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:09.590697 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:09.591508 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:09.591837 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:09.591944 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:09.592887 (+ 943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:43:09.592891 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:43:09.592893 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:43:09.592907 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:43:09.592939 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:43:09.592952 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:43:09.593039 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:09.593190 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 04:43:09.593230 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 04:43:09.593494 21514 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:43:09.594357 21514 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:43:09.599900 21514 catalog_manager.cc:1347] Generated new cluster ID: b20414da0c814b448a27545e9153c6a9
I20260925 04:43:09.599992 21514 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:43:09.625842 21514 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:43:09.626835 21514 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:43:09.630867 21514 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b37c4cee696742f69da8d6501fc6d2ac: Generated new TSK 0
I20260925 04:43:09.631574 21514 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:43:09.639560 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 21427
W20260925 04:43:09.644528 21412 connection.cc:535] client connection to 127.0.0.1:44345 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44345): Connection reset by peer (error 104)
2026-09-25T04:43:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1371 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790311123478858-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:43:09 dist-test-slave-dc95 krb5kdc[21526](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:43:09 dist-test-slave-dc95 krb5kdc[21526](info): set up 2 sockets
Sep 25 04:43:09 dist-test-slave-dc95 krb5kdc[21526](info): commencing operation
krb5kdc: starting...
W20260925 04:43:10.599694 392 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790311123478858-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790311123478858-392-0/krb5kdc/test-user.keytab.
Sep 25 04:43:10 dist-test-slave-dc95 krb5kdc[21526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:43:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:43:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790311123478858-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790311123478858-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 04:43:10.723785 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790311123478858-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790311123478858-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790311123478858-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37463
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790311123478858-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37463
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:43:10.841197 21539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:43:10.841509 21539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:43:10.841579 21539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:43:10.845945 21539 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:43:10.846035 21539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:43:10.846062 21539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:43:10.846086 21539 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:43:10.846117 21539 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:43:10.851786 21539 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790311123478858-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790311123478858-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37463
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790311123478858-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790311123478858-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:43:10.852946 21539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:43:10.853987 21539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:43:10.862844 21544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:43:10.862843 21547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:43:10.863184 21545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:43:10.862998 21539 server_base.cc:1034] running on GCE node
I20260925 04:43:10.863909 21539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:43:10.865119 21539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:43:10.866302 21539 hybrid_clock.cc:648] HybridClock initialized: now 1790311390866274 us; error 43 us; skew 500 ppm
Sep 25 04:43:10 dist-test-slave-dc95 krb5kdc[21526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311390, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:43:10.869400 21539 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 04:43:10.870386 21539 webserver.cc:460] Webserver started at http://127.0.0.1:44389/ using document root <none> and password file <none>
I20260925 04:43:10.870975 21539 fs_manager.cc:359] Metadata directory not provided
I20260925 04:43:10.871048 21539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:43:10.871256 21539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:43:10.873476 21539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790311123478858-392-0/minicluster-data/master-0/data/instance:
uuid: "11c4f23c03784ae599ed693dafc061ac"
format_stamp: "Formatted at 2026-09-25 04:43:10 on dist-test-slave-dc95"
server_key: "daf12af45949c51e9a77fee84525647b"
server_key_iv: "debcc6e02fcc50e04658f8c1bd83352e"
server_key_version: "encryptionkey@0"
I20260925 04:43:10.874045 21539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790311123478858-392-0/minicluster-data/master-0/wal/instance:
uuid: "11c4f23c03784ae599ed693dafc061ac"
format_stamp: "Formatted at 2026-09-25 04:43:10 on dist-test-slave-dc95"
server_key: "daf12af45949c51e9a77fee84525647b"
server_key_iv: "debcc6e02fcc50e04658f8c1bd83352e"
server_key_version: "encryptionkey@0"
I20260925 04:43:10.877985 21539 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260925 04:43:10.880733 21554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:43:10.882184 21539 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:43:10.882330 21539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790311123478858-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790311123478858-392-0/minicluster-data/master-0/wal
uuid: "11c4f23c03784ae599ed693dafc061ac"
format_stamp: "Formatted at 2026-09-25 04:43:10 on dist-test-slave-dc95"
server_key: "daf12af45949c51e9a77fee84525647b"
server_key_iv: "debcc6e02fcc50e04658f8c1bd83352e"
server_key_version: "encryptionkey@0"
I20260925 04:43:10.882463 21539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790311123478858-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790311123478858-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790311123478858-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:43:10.887377 21539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:43:10.890893 21539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:43:10.891100 21539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:43:10.898365 21539 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37463
I20260925 04:43:10.898422 21606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37463 every 8 connection(s)
I20260925 04:43:10.899336 21539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
I20260925 04:43:10.900108 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 21539
I20260925 04:43:10.900249 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790311123478858-392-0/minicluster-data/master-0/wal/instance
I20260925 04:43:10.900538 392 external_mini_cluster.cc:1428] Setting key f0db00de7363ef34b05dd4c26f0f4e51
I20260925 04:43:10.902827 21607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:43:10.906461 21610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:10.901908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:57135 (local address 127.0.0.1:37463)
0925 04:43:10.902337 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:10.902355 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:10.902397 (+ 42us) server_negotiation.cc:407] Connection header received
0925 04:43:10.903253 (+ 856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:10.903278 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:10.903685 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:10.904076 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:10.904814 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:43:10.904820 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:43:10.904835 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:43:10.904846 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:43:10.904906 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:43:10.904932 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:43:10.905086 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:10.905155 (+ 69us) server_negotiation.cc:299] Negotiation successful
0925 04:43:10.905321 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":125,"threads_started":1}
I20260925 04:43:10.909852 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:43:10.910537 21607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11c4f23c03784ae599ed693dafc061ac: Bootstrap starting.
I20260925 04:43:10.913080 21607 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11c4f23c03784ae599ed693dafc061ac: Neither blocks nor log segments found. Creating new log.
I20260925 04:43:10.913568 21610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:10.911899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:48505 (local address 127.0.0.1:37463)
0925 04:43:10.912034 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:10.912039 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:10.912163 (+ 124us) server_negotiation.cc:407] Connection header received
0925 04:43:10.912271 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:10.912274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:10.912315 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:10.912403 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:10.913195 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:43:10.913199 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:43:10.913202 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:43:10.913214 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:43:10.913245 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:43:10.913258 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:43:10.913355 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:10.913408 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 04:43:10.913448 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 04:43:10.913955 21607 log.cc:824] T 00000000000000000000000000000000 P 11c4f23c03784ae599ed693dafc061ac: Log is configured to *not* fsync() on all Append() calls
I20260925 04:43:10.916069 21607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11c4f23c03784ae599ed693dafc061ac: No bootstrap required, opened a new log
I20260925 04:43:10.919677 21607 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11c4f23c03784ae599ed693dafc061ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11c4f23c03784ae599ed693dafc061ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37463 } }
I20260925 04:43:10.919941 21607 raft_consensus.cc:374] T 00000000000000000000000000000000 P 11c4f23c03784ae599ed693dafc061ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:43:10.920001 21607 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11c4f23c03784ae599ed693dafc061ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11c4f23c03784ae599ed693dafc061ac, State: Initialized, Role: FOLLOWER
I20260925 04:43:10.920478 21607 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11c4f23c03784ae599ed693dafc061ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11c4f23c03784ae599ed693dafc061ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37463 } }
I20260925 04:43:10.920696 21607 raft_consensus.cc:388] T 00000000000000000000000000000000 P 11c4f23c03784ae599ed693dafc061ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:43:10.920771 21607 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11c4f23c03784ae599ed693dafc061ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:43:10.920867 21607 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11c4f23c03784ae599ed693dafc061ac [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:43:10.921823 21607 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11c4f23c03784ae599ed693dafc061ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11c4f23c03784ae599ed693dafc061ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37463 } }
I20260925 04:43:10.922158 21607 leader_election.cc:302] T 00000000000000000000000000000000 P 11c4f23c03784ae599ed693dafc061ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 11c4f23c03784ae599ed693dafc061ac; no voters:
I20260925 04:43:10.922454 21607 leader_election.cc:288] T 00000000000000000000000000000000 P 11c4f23c03784ae599ed693dafc061ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:43:10.922580 21617 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11c4f23c03784ae599ed693dafc061ac [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:43:10.922801 21617 raft_consensus.cc:686] T 00000000000000000000000000000000 P 11c4f23c03784ae599ed693dafc061ac [term 1 LEADER]: Becoming Leader. State: Replica: 11c4f23c03784ae599ed693dafc061ac, State: Running, Role: LEADER
I20260925 04:43:10.923184 21617 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11c4f23c03784ae599ed693dafc061ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11c4f23c03784ae599ed693dafc061ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37463 } }
I20260925 04:43:10.923686 21607 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11c4f23c03784ae599ed693dafc061ac [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:43:10.924768 21618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11c4f23c03784ae599ed693dafc061ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "11c4f23c03784ae599ed693dafc061ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11c4f23c03784ae599ed693dafc061ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37463 } } }
I20260925 04:43:10.924875 21618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11c4f23c03784ae599ed693dafc061ac [sys.catalog]: This master's current role is: LEADER
I20260925 04:43:10.925168 21619 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11c4f23c03784ae599ed693dafc061ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11c4f23c03784ae599ed693dafc061ac. Latest consensus state: current_term: 1 leader_uuid: "11c4f23c03784ae599ed693dafc061ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11c4f23c03784ae599ed693dafc061ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37463 } } }
I20260925 04:43:10.925249 21626 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:43:10.925323 21619 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11c4f23c03784ae599ed693dafc061ac [sys.catalog]: This master's current role is: LEADER
I20260925 04:43:10.928496 21626 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:43:10.933995 21626 catalog_manager.cc:1347] Generated new cluster ID: 4905aa53bdb840bc9af72c0c68105b6b
I20260925 04:43:10.934101 21626 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:43:10.990012 21626 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:43:10.990391 21626 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:43:10.996438 21626 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 11c4f23c03784ae599ed693dafc061ac: Generated new TSK 0
I20260925 04:43:10.997246 21626 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:43:11.006572 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 21539
2026-09-25T04:43:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1366 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790311123478858-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:43:11 dist-test-slave-dc95 krb5kdc[21638](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:43:11 dist-test-slave-dc95 krb5kdc[21638](info): set up 2 sockets
Sep 25 04:43:11 dist-test-slave-dc95 krb5kdc[21638](info): commencing operation
krb5kdc: starting...
W20260925 04:43:11.963646 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790311123478858-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790311123478858-392-0/krb5kdc/test-user.keytab.
Sep 25 04:43:12 dist-test-slave-dc95 krb5kdc[21638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:43:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:43:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790311123478858-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790311123478858-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 04:43:12.095201 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790311123478858-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790311123478858-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790311123478858-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36973
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790311123478858-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36973
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:43:12.209357 21651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:43:12.209662 21651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:43:12.209705 21651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:43:12.214154 21651 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:43:12.214241 21651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:43:12.214260 21651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:43:12.214277 21651 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:43:12.214291 21651 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:43:12.220501 21651 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790311123478858-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790311123478858-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36973
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790311123478858-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790311123478858-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:43:12.221666 21651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:43:12.222700 21651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:43:12.231861 21656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:43:12.231861 21657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:43:12.231917 21651 server_base.cc:1034] running on GCE node
W20260925 04:43:12.231864 21659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:43:12.232684 21651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:43:12.233868 21651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:43:12.235054 21651 hybrid_clock.cc:648] HybridClock initialized: now 1790311392235036 us; error 31 us; skew 500 ppm
Sep 25 04:43:12 dist-test-slave-dc95 krb5kdc[21638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311392, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:43:12.237908 21651 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 04:43:12.238919 21651 webserver.cc:460] Webserver started at http://127.0.0.1:35137/ using document root <none> and password file <none>
I20260925 04:43:12.239471 21651 fs_manager.cc:359] Metadata directory not provided
I20260925 04:43:12.239540 21651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:43:12.239768 21651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:43:12.241840 21651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790311123478858-392-0/minicluster-data/master-0/data/instance:
uuid: "d030891738c145519c3e8bff469508d5"
format_stamp: "Formatted at 2026-09-25 04:43:12 on dist-test-slave-dc95"
server_key: "14d2abe4b1bc74306e75eda7458e9201"
server_key_iv: "3e6ce20e89ec9f2733757e4d61c06598"
server_key_version: "encryptionkey@0"
I20260925 04:43:12.242381 21651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790311123478858-392-0/minicluster-data/master-0/wal/instance:
uuid: "d030891738c145519c3e8bff469508d5"
format_stamp: "Formatted at 2026-09-25 04:43:12 on dist-test-slave-dc95"
server_key: "14d2abe4b1bc74306e75eda7458e9201"
server_key_iv: "3e6ce20e89ec9f2733757e4d61c06598"
server_key_version: "encryptionkey@0"
I20260925 04:43:12.246212 21651 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 04:43:12.248874 21666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:43:12.250178 21651 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 04:43:12.250300 21651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790311123478858-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790311123478858-392-0/minicluster-data/master-0/wal
uuid: "d030891738c145519c3e8bff469508d5"
format_stamp: "Formatted at 2026-09-25 04:43:12 on dist-test-slave-dc95"
server_key: "14d2abe4b1bc74306e75eda7458e9201"
server_key_iv: "3e6ce20e89ec9f2733757e4d61c06598"
server_key_version: "encryptionkey@0"
I20260925 04:43:12.250429 21651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790311123478858-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790311123478858-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790311123478858-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:43:12.268422 21651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:43:12.271675 21651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:43:12.271876 21651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:43:12.279661 21651 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36973
I20260925 04:43:12.279696 21718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36973 every 8 connection(s)
I20260925 04:43:12.280700 21651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
I20260925 04:43:12.281929 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 21651
I20260925 04:43:12.282049 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790311123478858-392-0/minicluster-data/master-0/wal/instance
I20260925 04:43:12.282469 392 external_mini_cluster.cc:1428] Setting key 3ef881ce9b965e1a445fc78d6fa4b82b
I20260925 04:43:12.284148 21719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:43:12.290169 21719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d030891738c145519c3e8bff469508d5: Bootstrap starting.
Sep 25 04:43:12 dist-test-slave-dc95 krb5kdc[21638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 04:43:12.292695 21719 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d030891738c145519c3e8bff469508d5: Neither blocks nor log segments found. Creating new log.
I20260925 04:43:12.293519 21719 log.cc:824] T 00000000000000000000000000000000 P d030891738c145519c3e8bff469508d5: Log is configured to *not* fsync() on all Append() calls
I20260925 04:43:12.295929 21719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d030891738c145519c3e8bff469508d5: No bootstrap required, opened a new log
I20260925 04:43:12.297302 21722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:12.283985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:48673 (local address 127.0.0.1:36973)
0925 04:43:12.284501 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:12.284517 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:12.284554 (+ 37us) server_negotiation.cc:407] Connection header received
0925 04:43:12.285267 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:12.285287 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:12.285669 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:12.286002 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:12.286812 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:12.287911 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:12.288798 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:12.289312 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:12.289474 (+ 162us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:12.292049 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:43:12.292055 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:43:12.292071 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:43:12.292106 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:43:12.294208 (+ 2102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:12.294846 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:12.294851 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:12.294856 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:12.294972 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:12.295295 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:12.295298 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:12.295299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:12.295752 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:43:12.295911 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:12.296255 (+ 344us) server_negotiation.cc:299] Negotiation successful
0925 04:43:12.296492 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":153,"threads_started":1}
I20260925 04:43:12.299957 21719 raft_consensus.cc:348] T 00000000000000000000000000000000 P d030891738c145519c3e8bff469508d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d030891738c145519c3e8bff469508d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36973 } }
I20260925 04:43:12.300251 21719 raft_consensus.cc:374] T 00000000000000000000000000000000 P d030891738c145519c3e8bff469508d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:43:12.300331 21719 raft_consensus.cc:729] T 00000000000000000000000000000000 P d030891738c145519c3e8bff469508d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d030891738c145519c3e8bff469508d5, State: Initialized, Role: FOLLOWER
I20260925 04:43:12.300586 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:43:12.300781 21719 consensus_queue.cc:260] T 00000000000000000000000000000000 P d030891738c145519c3e8bff469508d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d030891738c145519c3e8bff469508d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36973 } }
I20260925 04:43:12.300923 21719 raft_consensus.cc:388] T 00000000000000000000000000000000 P d030891738c145519c3e8bff469508d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:43:12.300985 21719 raft_consensus.cc:482] T 00000000000000000000000000000000 P d030891738c145519c3e8bff469508d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:43:12.301079 21719 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d030891738c145519c3e8bff469508d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:43:12.302045 21719 raft_consensus.cc:504] T 00000000000000000000000000000000 P d030891738c145519c3e8bff469508d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d030891738c145519c3e8bff469508d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36973 } }
I20260925 04:43:12.302386 21719 leader_election.cc:302] T 00000000000000000000000000000000 P d030891738c145519c3e8bff469508d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d030891738c145519c3e8bff469508d5; no voters:
I20260925 04:43:12.302719 21719 leader_election.cc:288] T 00000000000000000000000000000000 P d030891738c145519c3e8bff469508d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:43:12.303097 21730 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d030891738c145519c3e8bff469508d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:43:12.303279 21730 raft_consensus.cc:686] T 00000000000000000000000000000000 P d030891738c145519c3e8bff469508d5 [term 1 LEADER]: Becoming Leader. State: Replica: d030891738c145519c3e8bff469508d5, State: Running, Role: LEADER
I20260925 04:43:12.303586 21730 consensus_queue.cc:237] T 00000000000000000000000000000000 P d030891738c145519c3e8bff469508d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d030891738c145519c3e8bff469508d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36973 } }
I20260925 04:43:12.304147 21719 sys_catalog.cc:564] T 00000000000000000000000000000000 P d030891738c145519c3e8bff469508d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:43:12.304864 21730 sys_catalog.cc:455] T 00000000000000000000000000000000 P d030891738c145519c3e8bff469508d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d030891738c145519c3e8bff469508d5. Latest consensus state: current_term: 1 leader_uuid: "d030891738c145519c3e8bff469508d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d030891738c145519c3e8bff469508d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36973 } } }
I20260925 04:43:12.304997 21730 sys_catalog.cc:458] T 00000000000000000000000000000000 P d030891738c145519c3e8bff469508d5 [sys.catalog]: This master's current role is: LEADER
I20260925 04:43:12.305361 21729 sys_catalog.cc:455] T 00000000000000000000000000000000 P d030891738c145519c3e8bff469508d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d030891738c145519c3e8bff469508d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d030891738c145519c3e8bff469508d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36973 } } }
I20260925 04:43:12.305477 21729 sys_catalog.cc:458] T 00000000000000000000000000000000 P d030891738c145519c3e8bff469508d5 [sys.catalog]: This master's current role is: LEADER
W20260925 04:43:12.308990 21744 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d030891738c145519c3e8bff469508d5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 04:43:12.309078 21744 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 04:43:12.309141 21738 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:43:12.309971 21738 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:43:12.311394 21722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:12.302616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:35319 (local address 127.0.0.1:36973)
0925 04:43:12.302729 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:12.302733 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:12.302951 (+ 218us) server_negotiation.cc:407] Connection header received
0925 04:43:12.303061 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:12.303064 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:12.303115 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:12.303239 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:12.304139 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:12.304763 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:12.305913 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:12.306251 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:12.306806 (+ 555us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:12.308085 (+ 1279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:43:12.308089 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:43:12.308092 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:43:12.308117 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:43:12.309495 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:12.310052 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:12.310056 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:12.310058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:12.310149 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:12.310543 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:12.310546 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:12.310547 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:12.310731 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:43:12.310906 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:12.311171 (+ 265us) server_negotiation.cc:299] Negotiation successful
0925 04:43:12.311259 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 04:43:12.316381 21738 catalog_manager.cc:1347] Generated new cluster ID: 13bbfc6a468c4cbbb5d57ed829b882d2
I20260925 04:43:12.316473 21738 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:43:12.329873 21738 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:43:12.330931 21738 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:43:12.336472 21738 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d030891738c145519c3e8bff469508d5: Generated new TSK 0
I20260925 04:43:12.337381 21738 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:43:12.347370 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 21651
W20260925 04:43:12.354209 21636 connection.cc:535] client connection to 127.0.0.1:36973 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36973): Connection reset by peer (error 104)
2026-09-25T04:43:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1342 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790311123478858-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:43:12 dist-test-slave-dc95 krb5kdc[21750](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:43:12 dist-test-slave-dc95 krb5kdc[21750](info): set up 2 sockets
Sep 25 04:43:12 dist-test-slave-dc95 krb5kdc[21750](info): commencing operation
krb5kdc: starting...
W20260925 04:43:13.327732 392 mini_kdc.cc:121] Time spent starting KDC: real 0.959s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790311123478858-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790311123478858-392-0/krb5kdc/test-user.keytab.
Sep 25 04:43:13 dist-test-slave-dc95 krb5kdc[21750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:43:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:43:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790311123478858-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790311123478858-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 04:43:13.458979 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790311123478858-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790311123478858-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790311123478858-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45855
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790311123478858-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45855
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:43:13.574438 21763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:43:13.574755 21763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:43:13.574819 21763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:43:13.579203 21763 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:43:13.579303 21763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:43:13.579331 21763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:43:13.579352 21763 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:43:13.579372 21763 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:43:13.585403 21763 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790311123478858-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790311123478858-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45855
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790311123478858-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790311123478858-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:43:13.586822 21763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:43:13.588080 21763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 04:43:13.597427 21763 server_base.cc:1034] running on GCE node
W20260925 04:43:13.597383 21768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:43:13.597383 21771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:43:13.597388 21769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:43:13.598273 21763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:43:13.599483 21763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:43:13.600706 21763 hybrid_clock.cc:648] HybridClock initialized: now 1790311393600692 us; error 29 us; skew 500 ppm
Sep 25 04:43:13 dist-test-slave-dc95 krb5kdc[21750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311393, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:43:13.603729 21763 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 04:43:13.604743 21763 webserver.cc:460] Webserver started at http://127.0.0.1:41949/ using document root <none> and password file <none>
I20260925 04:43:13.605340 21763 fs_manager.cc:359] Metadata directory not provided
I20260925 04:43:13.605409 21763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:43:13.605614 21763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:43:13.607839 21763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790311123478858-392-0/minicluster-data/master-0/data/instance:
uuid: "ea03995747544db2bbe6a7a78344f858"
format_stamp: "Formatted at 2026-09-25 04:43:13 on dist-test-slave-dc95"
server_key: "ed2bd3af82fb60a2791fa09f46f0b251"
server_key_iv: "b5ed0478e3d8e72e18cc5d5bbe41536c"
server_key_version: "encryptionkey@0"
I20260925 04:43:13.608428 21763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790311123478858-392-0/minicluster-data/master-0/wal/instance:
uuid: "ea03995747544db2bbe6a7a78344f858"
format_stamp: "Formatted at 2026-09-25 04:43:13 on dist-test-slave-dc95"
server_key: "ed2bd3af82fb60a2791fa09f46f0b251"
server_key_iv: "b5ed0478e3d8e72e18cc5d5bbe41536c"
server_key_version: "encryptionkey@0"
I20260925 04:43:13.612782 21763 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:43:13.615679 21778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:43:13.617146 21763 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:43:13.617352 21763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790311123478858-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790311123478858-392-0/minicluster-data/master-0/wal
uuid: "ea03995747544db2bbe6a7a78344f858"
format_stamp: "Formatted at 2026-09-25 04:43:13 on dist-test-slave-dc95"
server_key: "ed2bd3af82fb60a2791fa09f46f0b251"
server_key_iv: "b5ed0478e3d8e72e18cc5d5bbe41536c"
server_key_version: "encryptionkey@0"
I20260925 04:43:13.617483 21763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790311123478858-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790311123478858-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790311123478858-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:43:13.635322 21763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:43:13.638732 21763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:43:13.638943 21763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:43:13.646040 21763 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45855
I20260925 04:43:13.646067 21830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45855 every 8 connection(s)
I20260925 04:43:13.647053 21763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
I20260925 04:43:13.650002 21831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:43:13.656190 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 21763
I20260925 04:43:13.656348 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790311123478858-392-0/minicluster-data/master-0/wal/instance
I20260925 04:43:13.656631 392 external_mini_cluster.cc:1428] Setting key c701f985a8d14a8853358ab56cda987b
I20260925 04:43:13.657462 21831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea03995747544db2bbe6a7a78344f858: Bootstrap starting.
I20260925 04:43:13.660181 21831 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea03995747544db2bbe6a7a78344f858: Neither blocks nor log segments found. Creating new log.
I20260925 04:43:13.660980 21831 log.cc:824] T 00000000000000000000000000000000 P ea03995747544db2bbe6a7a78344f858: Log is configured to *not* fsync() on all Append() calls
I20260925 04:43:13.663084 21831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea03995747544db2bbe6a7a78344f858: No bootstrap required, opened a new log
I20260925 04:43:13.667073 21831 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea03995747544db2bbe6a7a78344f858 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea03995747544db2bbe6a7a78344f858" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45855 } }
I20260925 04:43:13.667358 21831 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea03995747544db2bbe6a7a78344f858 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:43:13.667418 21831 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea03995747544db2bbe6a7a78344f858 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea03995747544db2bbe6a7a78344f858, State: Initialized, Role: FOLLOWER
I20260925 04:43:13.667922 21834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:13.658253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:52573 (local address 127.0.0.1:45855)
0925 04:43:13.658782 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:13.658798 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:13.658839 (+ 41us) server_negotiation.cc:407] Connection header received
0925 04:43:13.659598 (+ 759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:13.659685 (+ 87us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:13.660088 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:13.660469 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:13.660943 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:13.662107 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:13.663580 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:13.664450 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:13.664802 (+ 352us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:13.665888 (+ 1086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:43:13.665893 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:43:13.665910 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:43:13.666410 (+ 500us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:43:13.666476 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:43:13.666504 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:43:13.666644 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:13.666838 (+ 194us) server_negotiation.cc:299] Negotiation successful
0925 04:43:13.666989 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":156,"threads_started":1}
I20260925 04:43:13.667908 21831 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea03995747544db2bbe6a7a78344f858 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea03995747544db2bbe6a7a78344f858" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45855 } }
I20260925 04:43:13.668088 21831 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea03995747544db2bbe6a7a78344f858 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:43:13.668171 21831 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea03995747544db2bbe6a7a78344f858 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:43:13.668267 21831 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea03995747544db2bbe6a7a78344f858 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:43:13.669450 21831 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea03995747544db2bbe6a7a78344f858 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea03995747544db2bbe6a7a78344f858" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45855 } }
I20260925 04:43:13.669816 21831 leader_election.cc:302] T 00000000000000000000000000000000 P ea03995747544db2bbe6a7a78344f858 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea03995747544db2bbe6a7a78344f858; no voters:
I20260925 04:43:13.670151 21831 leader_election.cc:288] T 00000000000000000000000000000000 P ea03995747544db2bbe6a7a78344f858 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:43:13.670279 21836 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea03995747544db2bbe6a7a78344f858 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:43:13.670480 21836 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea03995747544db2bbe6a7a78344f858 [term 1 LEADER]: Becoming Leader. State: Replica: ea03995747544db2bbe6a7a78344f858, State: Running, Role: LEADER
I20260925 04:43:13.670799 21836 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea03995747544db2bbe6a7a78344f858 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea03995747544db2bbe6a7a78344f858" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45855 } }
I20260925 04:43:13.671294 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:43:13.671526 21831 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea03995747544db2bbe6a7a78344f858 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:43:13.672536 21837 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea03995747544db2bbe6a7a78344f858 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea03995747544db2bbe6a7a78344f858" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea03995747544db2bbe6a7a78344f858" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45855 } } }
I20260925 04:43:13.672695 21837 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea03995747544db2bbe6a7a78344f858 [sys.catalog]: This master's current role is: LEADER
I20260925 04:43:13.672978 21838 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea03995747544db2bbe6a7a78344f858 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea03995747544db2bbe6a7a78344f858. Latest consensus state: current_term: 1 leader_uuid: "ea03995747544db2bbe6a7a78344f858" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea03995747544db2bbe6a7a78344f858" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45855 } } }
I20260925 04:43:13.673074 21838 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea03995747544db2bbe6a7a78344f858 [sys.catalog]: This master's current role is: LEADER
I20260925 04:43:13.673367 21849 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:43:13.676569 21849 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:43:13.679970 21834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:13.674793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:48425 (local address 127.0.0.1:45855)
0925 04:43:13.674928 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:13.674933 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:13.675222 (+ 289us) server_negotiation.cc:407] Connection header received
0925 04:43:13.675362 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:13.675365 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:13.675416 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:13.675492 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:13.675863 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:13.676662 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:13.677814 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:13.678146 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:13.678271 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:13.679274 (+ 1003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:43:13.679278 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:43:13.679281 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:43:13.679298 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:43:13.679332 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:43:13.679346 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:43:13.679431 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:13.679763 (+ 332us) server_negotiation.cc:299] Negotiation successful
0925 04:43:13.679811 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 04:43:13.682390 21849 catalog_manager.cc:1347] Generated new cluster ID: 79ab2055dd814712badd8c19377d5322
I20260925 04:43:13.682473 21849 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:43:13.693224 21849 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:43:13.694427 21849 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:43:13.701989 21849 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea03995747544db2bbe6a7a78344f858: Generated new TSK 0
I20260925 04:43:13.702726 21849 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:43:13.712435 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 21763
W20260925 04:43:13.717139 21748 connection.cc:535] client connection to 127.0.0.1:45855 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45855): Connection reset by peer (error 104)
2026-09-25T04:43:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1362 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790311123478858-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:43:13 dist-test-slave-dc95 krb5kdc[21862](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:43:13 dist-test-slave-dc95 krb5kdc[21862](info): set up 2 sockets
Sep 25 04:43:13 dist-test-slave-dc95 krb5kdc[21862](info): commencing operation
krb5kdc: starting...
W20260925 04:43:14.667666 392 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790311123478858-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790311123478858-392-0/krb5kdc/test-user.keytab.
Sep 25 04:43:14 dist-test-slave-dc95 krb5kdc[21862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311394, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:43:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:43:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790311123478858-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790311123478858-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 04:43:14.791316 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790311123478858-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790311123478858-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790311123478858-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43453
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790311123478858-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43453
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:43:14.913162 21875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:43:14.913450 21875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:43:14.913493 21875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:43:14.917662 21875 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:43:14.917733 21875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:43:14.917761 21875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:43:14.917776 21875 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:43:14.917791 21875 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:43:14.923852 21875 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790311123478858-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790311123478858-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43453
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790311123478858-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790311123478858-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:43:14.925371 21875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:43:14.926806 21875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:43:14.938071 21880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:43:14.938071 21883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:43:14.938064 21881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:43:14.938385 21875 server_base.cc:1034] running on GCE node
I20260925 04:43:14.938958 21875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:43:14.940169 21875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:43:14.941354 21875 hybrid_clock.cc:648] HybridClock initialized: now 1790311394941327 us; error 40 us; skew 500 ppm
Sep 25 04:43:14 dist-test-slave-dc95 krb5kdc[21862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311394, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:43:14.944171 21875 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 04:43:14.945158 21875 webserver.cc:460] Webserver started at http://127.0.0.1:41265/ using document root <none> and password file <none>
I20260925 04:43:14.945693 21875 fs_manager.cc:359] Metadata directory not provided
I20260925 04:43:14.945751 21875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:43:14.945921 21875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:43:14.948082 21875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790311123478858-392-0/minicluster-data/master-0/data/instance:
uuid: "98a2b3749c8648ffa73465f31db543e1"
format_stamp: "Formatted at 2026-09-25 04:43:14 on dist-test-slave-dc95"
server_key: "1095efe8d4b2a32a0ce04242c3673306"
server_key_iv: "cf2abed404784c74fcda71c8d1b39277"
server_key_version: "encryptionkey@0"
I20260925 04:43:14.948669 21875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790311123478858-392-0/minicluster-data/master-0/wal/instance:
uuid: "98a2b3749c8648ffa73465f31db543e1"
format_stamp: "Formatted at 2026-09-25 04:43:14 on dist-test-slave-dc95"
server_key: "1095efe8d4b2a32a0ce04242c3673306"
server_key_iv: "cf2abed404784c74fcda71c8d1b39277"
server_key_version: "encryptionkey@0"
I20260925 04:43:14.952528 21875 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 04:43:14.955017 21890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:43:14.956449 21875 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 04:43:14.956573 21875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790311123478858-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790311123478858-392-0/minicluster-data/master-0/wal
uuid: "98a2b3749c8648ffa73465f31db543e1"
format_stamp: "Formatted at 2026-09-25 04:43:14 on dist-test-slave-dc95"
server_key: "1095efe8d4b2a32a0ce04242c3673306"
server_key_iv: "cf2abed404784c74fcda71c8d1b39277"
server_key_version: "encryptionkey@0"
I20260925 04:43:14.956712 21875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790311123478858-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790311123478858-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790311123478858-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:43:14.961829 21875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:43:14.964874 21875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:43:14.965061 21875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:43:14.971803 21875 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43453
I20260925 04:43:14.971841 21942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43453 every 8 connection(s)
I20260925 04:43:14.972749 21875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
I20260925 04:43:14.975731 21943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:43:14.977660 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 21875
I20260925 04:43:14.977770 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790311123478858-392-0/minicluster-data/master-0/wal/instance
I20260925 04:43:14.978024 392 external_mini_cluster.cc:1428] Setting key 3abfc5c2fe98890026ca6868e94d192c
I20260925 04:43:14.982787 21943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98a2b3749c8648ffa73465f31db543e1: Bootstrap starting.
I20260925 04:43:14.983676 21946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:14.979404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:33393 (local address 127.0.0.1:43453)
0925 04:43:14.979876 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:14.979895 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:14.979938 (+ 43us) server_negotiation.cc:407] Connection header received
0925 04:43:14.980723 (+ 785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:14.980750 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:14.981144 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:14.981529 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:14.982199 (+ 670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:43:14.982204 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:43:14.982219 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:43:14.982230 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:43:14.982289 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:43:14.982313 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:43:14.982471 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:14.982540 (+ 69us) server_negotiation.cc:299] Negotiation successful
0925 04:43:14.982707 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":158,"threads_started":1}
I20260925 04:43:14.985419 21943 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98a2b3749c8648ffa73465f31db543e1: Neither blocks nor log segments found. Creating new log.
I20260925 04:43:14.986255 21943 log.cc:824] T 00000000000000000000000000000000 P 98a2b3749c8648ffa73465f31db543e1: Log is configured to *not* fsync() on all Append() calls
I20260925 04:43:14.986609 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:43:14.988761 21943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98a2b3749c8648ffa73465f31db543e1: No bootstrap required, opened a new log
I20260925 04:43:14.990701 21946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:14.988933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:44707 (local address 127.0.0.1:43453)
0925 04:43:14.989067 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:14.989071 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:14.989267 (+ 196us) server_negotiation.cc:407] Connection header received
0925 04:43:14.989391 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:14.989441 (+ 50us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:14.989475 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:14.989541 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:14.990301 (+ 760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:43:14.990304 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:43:14.990306 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:43:14.990318 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:43:14.990345 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:43:14.990355 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:43:14.990430 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:14.990505 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 04:43:14.990539 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 04:43:14.993072 21943 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98a2b3749c8648ffa73465f31db543e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98a2b3749c8648ffa73465f31db543e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43453 } }
I20260925 04:43:14.993336 21943 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98a2b3749c8648ffa73465f31db543e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:43:14.993374 21943 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98a2b3749c8648ffa73465f31db543e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98a2b3749c8648ffa73465f31db543e1, State: Initialized, Role: FOLLOWER
I20260925 04:43:14.993819 21943 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98a2b3749c8648ffa73465f31db543e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98a2b3749c8648ffa73465f31db543e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43453 } }
I20260925 04:43:14.993947 21943 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98a2b3749c8648ffa73465f31db543e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:43:14.994015 21943 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98a2b3749c8648ffa73465f31db543e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:43:14.994115 21943 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98a2b3749c8648ffa73465f31db543e1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:43:14.995199 21943 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98a2b3749c8648ffa73465f31db543e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98a2b3749c8648ffa73465f31db543e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43453 } }
I20260925 04:43:14.995553 21943 leader_election.cc:302] T 00000000000000000000000000000000 P 98a2b3749c8648ffa73465f31db543e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98a2b3749c8648ffa73465f31db543e1; no voters:
I20260925 04:43:14.995920 21943 leader_election.cc:288] T 00000000000000000000000000000000 P 98a2b3749c8648ffa73465f31db543e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:43:14.996043 21953 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98a2b3749c8648ffa73465f31db543e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:43:14.996253 21953 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98a2b3749c8648ffa73465f31db543e1 [term 1 LEADER]: Becoming Leader. State: Replica: 98a2b3749c8648ffa73465f31db543e1, State: Running, Role: LEADER
I20260925 04:43:14.996601 21953 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98a2b3749c8648ffa73465f31db543e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98a2b3749c8648ffa73465f31db543e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43453 } }
I20260925 04:43:14.997155 21943 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98a2b3749c8648ffa73465f31db543e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:43:14.998266 21954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98a2b3749c8648ffa73465f31db543e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "98a2b3749c8648ffa73465f31db543e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98a2b3749c8648ffa73465f31db543e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43453 } } }
I20260925 04:43:14.998288 21955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98a2b3749c8648ffa73465f31db543e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98a2b3749c8648ffa73465f31db543e1. Latest consensus state: current_term: 1 leader_uuid: "98a2b3749c8648ffa73465f31db543e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98a2b3749c8648ffa73465f31db543e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43453 } } }
I20260925 04:43:14.998445 21954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98a2b3749c8648ffa73465f31db543e1 [sys.catalog]: This master's current role is: LEADER
I20260925 04:43:14.998462 21955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98a2b3749c8648ffa73465f31db543e1 [sys.catalog]: This master's current role is: LEADER
I20260925 04:43:14.998911 21962 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:43:15.002156 21962 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:43:15.008149 21962 catalog_manager.cc:1347] Generated new cluster ID: 2f4934f2d52f4d28ba853848da85b591
I20260925 04:43:15.008236 21962 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:43:15.037442 21962 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:43:15.037822 21962 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:43:15.045451 21962 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98a2b3749c8648ffa73465f31db543e1: Generated new TSK 0
I20260925 04:43:15.046419 21962 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:43:15.059551 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 21875
2026-09-25T04:43:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1347 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790311123478858-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:43:15 dist-test-slave-dc95 krb5kdc[21974](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:43:15 dist-test-slave-dc95 krb5kdc[21974](info): set up 2 sockets
Sep 25 04:43:15 dist-test-slave-dc95 krb5kdc[21974](info): commencing operation
krb5kdc: starting...
W20260925 04:43:16.019776 392 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790311123478858-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790311123478858-392-0/krb5kdc/test-user.keytab.
Sep 25 04:43:16 dist-test-slave-dc95 krb5kdc[21974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:43:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:43:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 04:43:16.143229 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790311123478858-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790311123478858-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790311123478858-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38711
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38711
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:43:16.255697 21987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:43:16.256033 21987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:43:16.256098 21987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:43:16.260421 21987 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:43:16.260507 21987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:43:16.260535 21987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:43:16.260558 21987 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:43:16.260588 21987 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:43:16.266281 21987 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790311123478858-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790311123478858-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38711
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790311123478858-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:43:16.267485 21987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:43:16.268581 21987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:43:16.277794 21993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:43:16.277724 21995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:43:16.277724 21992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:43:16.277886 21987 server_base.cc:1034] running on GCE node
I20260925 04:43:16.278723 21987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:43:16.280130 21987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:43:16.281352 21987 hybrid_clock.cc:648] HybridClock initialized: now 1790311396281357 us; error 68 us; skew 500 ppm
Sep 25 04:43:16 dist-test-slave-dc95 krb5kdc[21974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311396, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:43:16.284554 21987 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 04:43:16.285567 21987 webserver.cc:460] Webserver started at http://127.0.98.62:40187/ using document root <none> and password file <none>
I20260925 04:43:16.286131 21987 fs_manager.cc:359] Metadata directory not provided
I20260925 04:43:16.286201 21987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:43:16.286408 21987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:43:16.288619 21987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790311123478858-392-0/minicluster-data/master-0/data/instance:
uuid: "3af20d7c7859446ab276b5c8a72f25f3"
format_stamp: "Formatted at 2026-09-25 04:43:16 on dist-test-slave-dc95"
server_key: "cce016df3ecaae5c56946c232daa168c"
server_key_iv: "66e028befbec14dff0d56d5ec3212d23"
server_key_version: "encryptionkey@0"
I20260925 04:43:16.289247 21987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790311123478858-392-0/minicluster-data/master-0/wal/instance:
uuid: "3af20d7c7859446ab276b5c8a72f25f3"
format_stamp: "Formatted at 2026-09-25 04:43:16 on dist-test-slave-dc95"
server_key: "cce016df3ecaae5c56946c232daa168c"
server_key_iv: "66e028befbec14dff0d56d5ec3212d23"
server_key_version: "encryptionkey@0"
I20260925 04:43:16.293672 21987 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 04:43:16.296618 22002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:43:16.298118 21987 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:43:16.298251 21987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790311123478858-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790311123478858-392-0/minicluster-data/master-0/wal
uuid: "3af20d7c7859446ab276b5c8a72f25f3"
format_stamp: "Formatted at 2026-09-25 04:43:16 on dist-test-slave-dc95"
server_key: "cce016df3ecaae5c56946c232daa168c"
server_key_iv: "66e028befbec14dff0d56d5ec3212d23"
server_key_version: "encryptionkey@0"
I20260925 04:43:16.298370 21987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790311123478858-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790311123478858-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790311123478858-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:43:16.353600 21987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:43:16.356947 21987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:43:16.357157 21987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:43:16.364380 21987 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38711
I20260925 04:43:16.364711 22054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38711 every 8 connection(s)
I20260925 04:43:16.365437 21987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
I20260925 04:43:16.368569 22055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:43:16.370870 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 21987
I20260925 04:43:16.371037 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790311123478858-392-0/minicluster-data/master-0/wal/instance
I20260925 04:43:16.371342 392 external_mini_cluster.cc:1428] Setting key e6ca3cf514e084767cbe460907803ca6
I20260925 04:43:16.375252 22055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3af20d7c7859446ab276b5c8a72f25f3: Bootstrap starting.
I20260925 04:43:16.377630 22055 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3af20d7c7859446ab276b5c8a72f25f3: Neither blocks nor log segments found. Creating new log.
I20260925 04:43:16.378398 22055 log.cc:824] T 00000000000000000000000000000000 P 3af20d7c7859446ab276b5c8a72f25f3: Log is configured to *not* fsync() on all Append() calls
Sep 25 04:43:16 dist-test-slave-dc95 krb5kdc[21974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:43:16.380735 22055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3af20d7c7859446ab276b5c8a72f25f3: No bootstrap required, opened a new log
I20260925 04:43:16.384483 22055 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3af20d7c7859446ab276b5c8a72f25f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3af20d7c7859446ab276b5c8a72f25f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38711 } }
I20260925 04:43:16.384732 22055 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3af20d7c7859446ab276b5c8a72f25f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:43:16.384785 22055 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3af20d7c7859446ab276b5c8a72f25f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3af20d7c7859446ab276b5c8a72f25f3, State: Initialized, Role: FOLLOWER
I20260925 04:43:16.385361 22055 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3af20d7c7859446ab276b5c8a72f25f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3af20d7c7859446ab276b5c8a72f25f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38711 } }
I20260925 04:43:16.385591 22060 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3af20d7c7859446ab276b5c8a72f25f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:43:16.385751 22060 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3af20d7c7859446ab276b5c8a72f25f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3af20d7c7859446ab276b5c8a72f25f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38711 } }
I20260925 04:43:16.385553 22055 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3af20d7c7859446ab276b5c8a72f25f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:43:16.386125 22055 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3af20d7c7859446ab276b5c8a72f25f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:43:16.386098 22060 leader_election.cc:302] T 00000000000000000000000000000000 P 3af20d7c7859446ab276b5c8a72f25f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3af20d7c7859446ab276b5c8a72f25f3; no voters:
I20260925 04:43:16.386191 22055 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3af20d7c7859446ab276b5c8a72f25f3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:43:16.386271 22058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:16.372861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39707 (local address 127.0.98.62:38711)
0925 04:43:16.373385 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:16.373402 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:16.373442 (+ 40us) server_negotiation.cc:407] Connection header received
0925 04:43:16.374289 (+ 847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:16.374309 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:16.374699 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:16.375046 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:16.375867 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:16.376871 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:16.377772 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:16.378260 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:16.378421 (+ 161us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:16.381264 (+ 2843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:43:16.381271 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:43:16.381288 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:43:16.381319 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:43:16.383067 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:16.383655 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:16.383660 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:16.383665 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:16.383790 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:16.384116 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:16.384118 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:16.384120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:16.384526 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:43:16.384746 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:16.385082 (+ 336us) server_negotiation.cc:299] Negotiation successful
0925 04:43:16.385317 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":149,"threads_started":1}
I20260925 04:43:16.386478 22060 leader_election.cc:288] T 00000000000000000000000000000000 P 3af20d7c7859446ab276b5c8a72f25f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers
I20260925 04:43:16.387490 22055 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3af20d7c7859446ab276b5c8a72f25f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3af20d7c7859446ab276b5c8a72f25f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38711 } }
I20260925 04:43:16.387663 22055 leader_election.cc:302] T 00000000000000000000000000000000 P 3af20d7c7859446ab276b5c8a72f25f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3af20d7c7859446ab276b5c8a72f25f3; no voters:
I20260925 04:43:16.387694 22061 raft_consensus.cc:2741] T 00000000000000000000000000000000 P 3af20d7c7859446ab276b5c8a72f25f3 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260925 04:43:16.387759 22055 leader_election.cc:288] T 00000000000000000000000000000000 P 3af20d7c7859446ab276b5c8a72f25f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:43:16.388027 22060 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3af20d7c7859446ab276b5c8a72f25f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:43:16.388269 22060 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3af20d7c7859446ab276b5c8a72f25f3 [term 1 LEADER]: Becoming Leader. State: Replica: 3af20d7c7859446ab276b5c8a72f25f3, State: Running, Role: LEADER
I20260925 04:43:16.388614 22060 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3af20d7c7859446ab276b5c8a72f25f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3af20d7c7859446ab276b5c8a72f25f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38711 } }
I20260925 04:43:16.389037 22055 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3af20d7c7859446ab276b5c8a72f25f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:43:16.389745 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:43:16.392068 22062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3af20d7c7859446ab276b5c8a72f25f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3af20d7c7859446ab276b5c8a72f25f3. Latest consensus state: current_term: 1 leader_uuid: "3af20d7c7859446ab276b5c8a72f25f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3af20d7c7859446ab276b5c8a72f25f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38711 } } }
I20260925 04:43:16.392107 22061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3af20d7c7859446ab276b5c8a72f25f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3af20d7c7859446ab276b5c8a72f25f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3af20d7c7859446ab276b5c8a72f25f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38711 } } }
I20260925 04:43:16.392408 22062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3af20d7c7859446ab276b5c8a72f25f3 [sys.catalog]: This master's current role is: LEADER
I20260925 04:43:16.392408 22061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3af20d7c7859446ab276b5c8a72f25f3 [sys.catalog]: This master's current role is: LEADER
W20260925 04:43:16.393847 22075 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3af20d7c7859446ab276b5c8a72f25f3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 04:43:16.393911 22075 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 04:43:16.400421 22058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:16.392707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56043 (local address 127.0.98.62:38711)
0925 04:43:16.392828 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:16.392833 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:16.393070 (+ 237us) server_negotiation.cc:407] Connection header received
0925 04:43:16.393169 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:16.393173 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:16.393221 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:16.393316 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:16.394020 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:16.394573 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:16.395381 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:16.395660 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:16.395755 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:16.397203 (+ 1448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:43:16.397206 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:43:16.397209 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:43:16.397234 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:43:16.398570 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:16.399078 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:16.399082 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:16.399084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:16.399160 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:16.399469 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:16.399472 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:16.399474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:16.399731 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:43:16.399870 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:16.400154 (+ 284us) server_negotiation.cc:299] Negotiation successful
0925 04:43:16.400250 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 04:43:16.400815 22082 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:43:16.401952 22082 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:43:16.408577 22082 catalog_manager.cc:1347] Generated new cluster ID: fab5aa6d0c3d4eddbdf4e118cf4d87a2
I20260925 04:43:16.408659 22082 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:43:16.418184 22082 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:43:16.419232 22082 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:43:16.430563 22082 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3af20d7c7859446ab276b5c8a72f25f3: Generated new TSK 0
I20260925 04:43:16.431368 22082 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:43:16.441435 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 21987
W20260925 04:43:16.447351 21972 connection.cc:535] client connection to 127.0.98.62:38711 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:38711): Connection reset by peer (error 104)
2026-09-25T04:43:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1382 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790311123478858-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:43:16 dist-test-slave-dc95 krb5kdc[22087](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:43:16 dist-test-slave-dc95 krb5kdc[22087](info): set up 2 sockets
Sep 25 04:43:16 dist-test-slave-dc95 krb5kdc[22087](info): commencing operation
krb5kdc: starting...
W20260925 04:43:17.403715 392 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790311123478858-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790311123478858-392-0/krb5kdc/test-user.keytab.
Sep 25 04:43:17 dist-test-slave-dc95 krb5kdc[22087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:43:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:43:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 04:43:17.526413 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790311123478858-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790311123478858-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790311123478858-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36609
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36609
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:43:17.636178 22100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:43:17.636479 22100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:43:17.636543 22100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:43:17.640695 22100 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:43:17.640779 22100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:43:17.640805 22100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:43:17.640828 22100 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:43:17.640857 22100 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:43:17.646471 22100 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790311123478858-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790311123478858-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36609
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790311123478858-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:43:17.647591 22100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:43:17.648653 22100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 04:43:17.657212 22100 server_base.cc:1034] running on GCE node
W20260925 04:43:17.657142 22105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:43:17.657179 22106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:43:17.657141 22108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:43:17.658013 22100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:43:17.659263 22100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:43:17.660449 22100 hybrid_clock.cc:648] HybridClock initialized: now 1790311397660397 us; error 65 us; skew 500 ppm
Sep 25 04:43:17 dist-test-slave-dc95 krb5kdc[22087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311397, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:43:17.663362 22100 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 04:43:17.664397 22100 webserver.cc:460] Webserver started at http://127.0.98.62:42533/ using document root <none> and password file <none>
I20260925 04:43:17.664961 22100 fs_manager.cc:359] Metadata directory not provided
I20260925 04:43:17.665027 22100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:43:17.665238 22100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:43:17.667322 22100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790311123478858-392-0/minicluster-data/master-0/data/instance:
uuid: "98849723381c4576ab5d16fd5e2fb59f"
format_stamp: "Formatted at 2026-09-25 04:43:17 on dist-test-slave-dc95"
server_key: "994dc40f0647cdc7ef5c051e9d26cad7"
server_key_iv: "56928183b3b16cbd818460bb96a073a8"
server_key_version: "encryptionkey@0"
I20260925 04:43:17.667918 22100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790311123478858-392-0/minicluster-data/master-0/wal/instance:
uuid: "98849723381c4576ab5d16fd5e2fb59f"
format_stamp: "Formatted at 2026-09-25 04:43:17 on dist-test-slave-dc95"
server_key: "994dc40f0647cdc7ef5c051e9d26cad7"
server_key_iv: "56928183b3b16cbd818460bb96a073a8"
server_key_version: "encryptionkey@0"
I20260925 04:43:17.671732 22100 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 04:43:17.674250 22115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:43:17.675566 22100 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:43:17.675768 22100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790311123478858-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790311123478858-392-0/minicluster-data/master-0/wal
uuid: "98849723381c4576ab5d16fd5e2fb59f"
format_stamp: "Formatted at 2026-09-25 04:43:17 on dist-test-slave-dc95"
server_key: "994dc40f0647cdc7ef5c051e9d26cad7"
server_key_iv: "56928183b3b16cbd818460bb96a073a8"
server_key_version: "encryptionkey@0"
I20260925 04:43:17.675895 22100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790311123478858-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790311123478858-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790311123478858-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:43:17.700829 22100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:43:17.704170 22100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:43:17.704372 22100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:43:17.711143 22100 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36609
I20260925 04:43:17.711160 22167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36609 every 8 connection(s)
I20260925 04:43:17.712162 22100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
I20260925 04:43:17.712921 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 22100
I20260925 04:43:17.713052 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790311123478858-392-0/minicluster-data/master-0/wal/instance
I20260925 04:43:17.713336 392 external_mini_cluster.cc:1428] Setting key b367ee252c6de7edc5762f34b70ce0fd
I20260925 04:43:17.715636 22168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:43:17.722014 22168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98849723381c4576ab5d16fd5e2fb59f: Bootstrap starting.
I20260925 04:43:17.723034 22171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:17.714729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47479 (local address 127.0.98.62:36609)
0925 04:43:17.715213 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:17.715228 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:17.715272 (+ 44us) server_negotiation.cc:407] Connection header received
0925 04:43:17.716009 (+ 737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:17.716030 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:17.716401 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:17.716767 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:17.717181 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:17.718198 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:17.719158 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:17.719921 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:17.720097 (+ 176us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:17.720989 (+ 892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:43:17.720996 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:43:17.721013 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:43:17.721533 (+ 520us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:43:17.721603 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:43:17.721632 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:43:17.721777 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:17.721974 (+ 197us) server_negotiation.cc:299] Negotiation successful
0925 04:43:17.722113 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":139,"threads_started":1}
I20260925 04:43:17.724541 22168 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98849723381c4576ab5d16fd5e2fb59f: Neither blocks nor log segments found. Creating new log.
I20260925 04:43:17.725332 22168 log.cc:824] T 00000000000000000000000000000000 P 98849723381c4576ab5d16fd5e2fb59f: Log is configured to *not* fsync() on all Append() calls
I20260925 04:43:17.726059 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:43:17.727865 22168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98849723381c4576ab5d16fd5e2fb59f: No bootstrap required, opened a new log
I20260925 04:43:17.732609 22171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:17.728350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56961 (local address 127.0.98.62:36609)
0925 04:43:17.728500 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:17.728505 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:17.728768 (+ 263us) server_negotiation.cc:407] Connection header received
0925 04:43:17.728941 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:17.728944 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:17.728988 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:17.729060 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:17.729365 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:17.729884 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:17.731002 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:17.731209 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:17.731284 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:17.732178 (+ 894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:43:17.732181 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:43:17.732183 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:43:17.732196 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:43:17.732222 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:43:17.732234 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:43:17.732306 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:17.732431 (+ 125us) server_negotiation.cc:299] Negotiation successful
0925 04:43:17.732462 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 04:43:17.732626 22168 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98849723381c4576ab5d16fd5e2fb59f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98849723381c4576ab5d16fd5e2fb59f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36609 } }
I20260925 04:43:17.732997 22168 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98849723381c4576ab5d16fd5e2fb59f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:43:17.733063 22168 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98849723381c4576ab5d16fd5e2fb59f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98849723381c4576ab5d16fd5e2fb59f, State: Initialized, Role: FOLLOWER
I20260925 04:43:17.733566 22168 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98849723381c4576ab5d16fd5e2fb59f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98849723381c4576ab5d16fd5e2fb59f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36609 } }
I20260925 04:43:17.733737 22168 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98849723381c4576ab5d16fd5e2fb59f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:43:17.733806 22168 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98849723381c4576ab5d16fd5e2fb59f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:43:17.733906 22168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98849723381c4576ab5d16fd5e2fb59f [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:43:17.734854 22168 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98849723381c4576ab5d16fd5e2fb59f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98849723381c4576ab5d16fd5e2fb59f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36609 } }
I20260925 04:43:17.735211 22168 leader_election.cc:302] T 00000000000000000000000000000000 P 98849723381c4576ab5d16fd5e2fb59f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98849723381c4576ab5d16fd5e2fb59f; no voters:
I20260925 04:43:17.735518 22168 leader_election.cc:288] T 00000000000000000000000000000000 P 98849723381c4576ab5d16fd5e2fb59f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:43:17.735662 22178 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98849723381c4576ab5d16fd5e2fb59f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:43:17.735848 22178 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98849723381c4576ab5d16fd5e2fb59f [term 1 LEADER]: Becoming Leader. State: Replica: 98849723381c4576ab5d16fd5e2fb59f, State: Running, Role: LEADER
I20260925 04:43:17.736196 22178 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98849723381c4576ab5d16fd5e2fb59f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98849723381c4576ab5d16fd5e2fb59f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36609 } }
I20260925 04:43:17.736769 22168 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98849723381c4576ab5d16fd5e2fb59f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:43:17.737833 22180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98849723381c4576ab5d16fd5e2fb59f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98849723381c4576ab5d16fd5e2fb59f. Latest consensus state: current_term: 1 leader_uuid: "98849723381c4576ab5d16fd5e2fb59f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98849723381c4576ab5d16fd5e2fb59f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36609 } } }
I20260925 04:43:17.737957 22180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98849723381c4576ab5d16fd5e2fb59f [sys.catalog]: This master's current role is: LEADER
I20260925 04:43:17.738251 22179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98849723381c4576ab5d16fd5e2fb59f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "98849723381c4576ab5d16fd5e2fb59f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98849723381c4576ab5d16fd5e2fb59f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36609 } } }
I20260925 04:43:17.738332 22179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98849723381c4576ab5d16fd5e2fb59f [sys.catalog]: This master's current role is: LEADER
W20260925 04:43:17.741523 22193 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 98849723381c4576ab5d16fd5e2fb59f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 04:43:17.741595 22193 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 04:43:17.741676 22187 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:43:17.742368 22187 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:43:17.748106 22187 catalog_manager.cc:1347] Generated new cluster ID: 9c7e1a696bd246579d4fe4d650b75f31
I20260925 04:43:17.748195 22187 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:43:17.757040 22187 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:43:17.758024 22187 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:43:17.762228 22187 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98849723381c4576ab5d16fd5e2fb59f: Generated new TSK 0
I20260925 04:43:17.762954 22187 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:43:17.772099 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 22100
W20260925 04:43:17.777099 22085 connection.cc:535] client connection to 127.0.98.62:36609 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:36609): Connection reset by peer (error 104)
2026-09-25T04:43:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1329 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790311123478858-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:43:17 dist-test-slave-dc95 krb5kdc[22199](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:43:17 dist-test-slave-dc95 krb5kdc[22199](info): set up 2 sockets
Sep 25 04:43:17 dist-test-slave-dc95 krb5kdc[22199](info): commencing operation
krb5kdc: starting...
W20260925 04:43:18.727690 392 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790311123478858-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790311123478858-392-0/krb5kdc/test-user.keytab.
Sep 25 04:43:18 dist-test-slave-dc95 krb5kdc[22199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311398, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:43:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:43:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 04:43:18.850349 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790311123478858-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790311123478858-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790311123478858-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35367
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:43:18.962714 22212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:43:18.963007 22212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:43:18.963066 22212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:43:18.967212 22212 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:43:18.967288 22212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:43:18.967315 22212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:43:18.967336 22212 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:43:18.967362 22212 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:43:18.972936 22212 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790311123478858-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790311123478858-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35367
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790311123478858-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:43:18.974076 22212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:43:18.975116 22212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:43:18.983771 22220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:43:18.983771 22217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:43:18.983806 22218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:43:18.983875 22212 server_base.cc:1034] running on GCE node
I20260925 04:43:18.984598 22212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:43:18.985728 22212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:43:18.986913 22212 hybrid_clock.cc:648] HybridClock initialized: now 1790311398986880 us; error 46 us; skew 500 ppm
Sep 25 04:43:18 dist-test-slave-dc95 krb5kdc[22199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311398, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:43:18.989652 22212 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 04:43:18.990554 22212 webserver.cc:460] Webserver started at http://127.0.98.62:32965/ using document root <none> and password file <none>
I20260925 04:43:18.991081 22212 fs_manager.cc:359] Metadata directory not provided
I20260925 04:43:18.991139 22212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:43:18.991304 22212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:43:18.993374 22212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790311123478858-392-0/minicluster-data/master-0/data/instance:
uuid: "81386cbb1ea3413e8b5191c9637abd47"
format_stamp: "Formatted at 2026-09-25 04:43:18 on dist-test-slave-dc95"
server_key: "b2f14a75fafce9ab9b6d4960b2b99a9a"
server_key_iv: "ab34dcdb5b7568ed31bba272e5e4ad23"
server_key_version: "encryptionkey@0"
I20260925 04:43:18.993906 22212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790311123478858-392-0/minicluster-data/master-0/wal/instance:
uuid: "81386cbb1ea3413e8b5191c9637abd47"
format_stamp: "Formatted at 2026-09-25 04:43:18 on dist-test-slave-dc95"
server_key: "b2f14a75fafce9ab9b6d4960b2b99a9a"
server_key_iv: "ab34dcdb5b7568ed31bba272e5e4ad23"
server_key_version: "encryptionkey@0"
I20260925 04:43:18.997725 22212 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260925 04:43:19.000295 22227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:43:19.001750 22212 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 04:43:19.001863 22212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790311123478858-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790311123478858-392-0/minicluster-data/master-0/wal
uuid: "81386cbb1ea3413e8b5191c9637abd47"
format_stamp: "Formatted at 2026-09-25 04:43:18 on dist-test-slave-dc95"
server_key: "b2f14a75fafce9ab9b6d4960b2b99a9a"
server_key_iv: "ab34dcdb5b7568ed31bba272e5e4ad23"
server_key_version: "encryptionkey@0"
I20260925 04:43:19.001968 22212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790311123478858-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790311123478858-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790311123478858-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:43:19.006673 22212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:43:19.009785 22212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:43:19.009944 22212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:43:19.017127 22212 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35367
I20260925 04:43:19.017184 22279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35367 every 8 connection(s)
I20260925 04:43:19.018139 22212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
I20260925 04:43:19.021248 22280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:43:19.026762 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 22212
I20260925 04:43:19.026896 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790311123478858-392-0/minicluster-data/master-0/wal/instance
I20260925 04:43:19.027177 392 external_mini_cluster.cc:1428] Setting key 98db605fd0d6c381b147634a9893b0b0
I20260925 04:43:19.027575 22280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81386cbb1ea3413e8b5191c9637abd47: Bootstrap starting.
I20260925 04:43:19.030033 22280 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81386cbb1ea3413e8b5191c9637abd47: Neither blocks nor log segments found. Creating new log.
I20260925 04:43:19.030900 22280 log.cc:824] T 00000000000000000000000000000000 P 81386cbb1ea3413e8b5191c9637abd47: Log is configured to *not* fsync() on all Append() calls
I20260925 04:43:19.032572 22283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:19.028638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35985 (local address 127.0.98.62:35367)
0925 04:43:19.029064 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:19.029079 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:19.029122 (+ 43us) server_negotiation.cc:407] Connection header received
0925 04:43:19.029789 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:19.029806 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:19.030138 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:19.030448 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:19.031254 (+ 806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:43:19.031258 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:43:19.031273 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:43:19.031284 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:43:19.031340 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:43:19.031369 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:43:19.031482 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:19.031629 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 04:43:19.031797 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":114,"threads_started":1}
I20260925 04:43:19.033236 22280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81386cbb1ea3413e8b5191c9637abd47: No bootstrap required, opened a new log
I20260925 04:43:19.035498 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:43:19.037402 22280 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81386cbb1ea3413e8b5191c9637abd47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81386cbb1ea3413e8b5191c9637abd47" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35367 } }
I20260925 04:43:19.037750 22280 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81386cbb1ea3413e8b5191c9637abd47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:43:19.037819 22280 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81386cbb1ea3413e8b5191c9637abd47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81386cbb1ea3413e8b5191c9637abd47, State: Initialized, Role: FOLLOWER
I20260925 04:43:19.038308 22280 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81386cbb1ea3413e8b5191c9637abd47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81386cbb1ea3413e8b5191c9637abd47" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35367 } }
I20260925 04:43:19.038477 22280 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81386cbb1ea3413e8b5191c9637abd47 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:43:19.038545 22280 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81386cbb1ea3413e8b5191c9637abd47 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:43:19.038647 22280 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81386cbb1ea3413e8b5191c9637abd47 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:43:19.039256 22283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:19.037501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36377 (local address 127.0.98.62:35367)
0925 04:43:19.037613 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:19.037618 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:19.037846 (+ 228us) server_negotiation.cc:407] Connection header received
0925 04:43:19.037948 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:19.037951 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:19.037988 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:19.038056 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:19.038776 (+ 720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:43:19.038780 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:43:19.038783 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:43:19.038795 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:43:19.038821 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:43:19.038832 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:43:19.038901 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:19.039074 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 04:43:19.039131 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 04:43:19.039815 22280 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81386cbb1ea3413e8b5191c9637abd47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81386cbb1ea3413e8b5191c9637abd47" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35367 } }
I20260925 04:43:19.040207 22280 leader_election.cc:302] T 00000000000000000000000000000000 P 81386cbb1ea3413e8b5191c9637abd47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81386cbb1ea3413e8b5191c9637abd47; no voters:
I20260925 04:43:19.040575 22280 leader_election.cc:288] T 00000000000000000000000000000000 P 81386cbb1ea3413e8b5191c9637abd47 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:43:19.040663 22290 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81386cbb1ea3413e8b5191c9637abd47 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:43:19.040848 22290 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81386cbb1ea3413e8b5191c9637abd47 [term 1 LEADER]: Becoming Leader. State: Replica: 81386cbb1ea3413e8b5191c9637abd47, State: Running, Role: LEADER
I20260925 04:43:19.041172 22290 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81386cbb1ea3413e8b5191c9637abd47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81386cbb1ea3413e8b5191c9637abd47" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35367 } }
I20260925 04:43:19.041975 22280 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81386cbb1ea3413e8b5191c9637abd47 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:43:19.043002 22291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81386cbb1ea3413e8b5191c9637abd47 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81386cbb1ea3413e8b5191c9637abd47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81386cbb1ea3413e8b5191c9637abd47" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35367 } } }
I20260925 04:43:19.043032 22292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81386cbb1ea3413e8b5191c9637abd47 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81386cbb1ea3413e8b5191c9637abd47. Latest consensus state: current_term: 1 leader_uuid: "81386cbb1ea3413e8b5191c9637abd47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81386cbb1ea3413e8b5191c9637abd47" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35367 } } }
I20260925 04:43:19.043217 22292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81386cbb1ea3413e8b5191c9637abd47 [sys.catalog]: This master's current role is: LEADER
I20260925 04:43:19.043490 22291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81386cbb1ea3413e8b5191c9637abd47 [sys.catalog]: This master's current role is: LEADER
I20260925 04:43:19.043632 22299 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:43:19.046916 22299 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:43:19.052870 22299 catalog_manager.cc:1347] Generated new cluster ID: a83e5534fb1641a19ecd978f47820df2
I20260925 04:43:19.052966 22299 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:43:19.064954 22299 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:43:19.065318 22299 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:43:19.070169 22299 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81386cbb1ea3413e8b5191c9637abd47: Generated new TSK 0
I20260925 04:43:19.070932 22299 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:43:19.078662 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 22212
2026-09-25T04:43:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1306 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790311123478858-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:43:19 dist-test-slave-dc95 krb5kdc[22311](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:43:19 dist-test-slave-dc95 krb5kdc[22311](info): set up 2 sockets
Sep 25 04:43:19 dist-test-slave-dc95 krb5kdc[22311](info): commencing operation
krb5kdc: starting...
W20260925 04:43:20.039736 392 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790311123478858-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790311123478858-392-0/krb5kdc/test-user.keytab.
Sep 25 04:43:20 dist-test-slave-dc95 krb5kdc[22311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:43:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:43:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 04:43:20.166534 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790311123478858-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790311123478858-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790311123478858-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35355
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35355
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:43:20.278285 22324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:43:20.278586 22324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:43:20.278630 22324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:43:20.282824 22324 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:43:20.282902 22324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:43:20.282920 22324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:43:20.282943 22324 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:43:20.282958 22324 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:43:20.288540 22324 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790311123478858-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790311123478858-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35355
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790311123478858-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:43:20.289613 22324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:43:20.290592 22324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 04:43:20.299553 22324 server_base.cc:1034] running on GCE node
W20260925 04:43:20.299497 22329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:43:20.299497 22332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:43:20.299528 22330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:43:20.300443 22324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:43:20.301705 22324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:43:20.302888 22324 hybrid_clock.cc:648] HybridClock initialized: now 1790311400302858 us; error 46 us; skew 500 ppm
Sep 25 04:43:20 dist-test-slave-dc95 krb5kdc[22311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:43:20.312912 22324 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 04:43:20.313967 22324 webserver.cc:460] Webserver started at http://127.0.98.62:34941/ using document root <none> and password file <none>
I20260925 04:43:20.314559 22324 fs_manager.cc:359] Metadata directory not provided
I20260925 04:43:20.314632 22324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:43:20.314846 22324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:43:20.317112 22324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790311123478858-392-0/minicluster-data/master-0/data/instance:
uuid: "ae07b292a70c4985bcf0f5c297af167c"
format_stamp: "Formatted at 2026-09-25 04:43:20 on dist-test-slave-dc95"
server_key: "2aca47e032caa315617ff9fea2ef4afc"
server_key_iv: "9e1f5202385e885064db40f48c448d9b"
server_key_version: "encryptionkey@0"
I20260925 04:43:20.317719 22324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790311123478858-392-0/minicluster-data/master-0/wal/instance:
uuid: "ae07b292a70c4985bcf0f5c297af167c"
format_stamp: "Formatted at 2026-09-25 04:43:20 on dist-test-slave-dc95"
server_key: "2aca47e032caa315617ff9fea2ef4afc"
server_key_iv: "9e1f5202385e885064db40f48c448d9b"
server_key_version: "encryptionkey@0"
I20260925 04:43:20.321935 22324 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 04:43:20.324898 22339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:43:20.326202 22324 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 04:43:20.326336 22324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790311123478858-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790311123478858-392-0/minicluster-data/master-0/wal
uuid: "ae07b292a70c4985bcf0f5c297af167c"
format_stamp: "Formatted at 2026-09-25 04:43:20 on dist-test-slave-dc95"
server_key: "2aca47e032caa315617ff9fea2ef4afc"
server_key_iv: "9e1f5202385e885064db40f48c448d9b"
server_key_version: "encryptionkey@0"
I20260925 04:43:20.326452 22324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790311123478858-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790311123478858-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790311123478858-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:43:20.341954 22324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:43:20.345069 22324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:43:20.345273 22324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:43:20.352370 22324 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35355
I20260925 04:43:20.352450 22391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35355 every 8 connection(s)
I20260925 04:43:20.353431 22324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
I20260925 04:43:20.356338 22392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:43:20.362099 22392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae07b292a70c4985bcf0f5c297af167c: Bootstrap starting.
I20260925 04:43:20.363214 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 22324
I20260925 04:43:20.363379 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790311123478858-392-0/minicluster-data/master-0/wal/instance
I20260925 04:43:20.363686 392 external_mini_cluster.cc:1428] Setting key 00e06dca18e0893f4b55d3d488c560d6
I20260925 04:43:20.365005 22392 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae07b292a70c4985bcf0f5c297af167c: Neither blocks nor log segments found. Creating new log.
I20260925 04:43:20.365808 22392 log.cc:824] T 00000000000000000000000000000000 P ae07b292a70c4985bcf0f5c297af167c: Log is configured to *not* fsync() on all Append() calls
I20260925 04:43:20.368111 22392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae07b292a70c4985bcf0f5c297af167c: No bootstrap required, opened a new log
I20260925 04:43:20.372607 22392 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae07b292a70c4985bcf0f5c297af167c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae07b292a70c4985bcf0f5c297af167c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35355 } }
I20260925 04:43:20.372892 22392 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae07b292a70c4985bcf0f5c297af167c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:43:20.372952 22392 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae07b292a70c4985bcf0f5c297af167c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae07b292a70c4985bcf0f5c297af167c, State: Initialized, Role: FOLLOWER
Sep 25 04:43:20 dist-test-slave-dc95 krb5kdc[22311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:43:20.373423 22392 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae07b292a70c4985bcf0f5c297af167c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae07b292a70c4985bcf0f5c297af167c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35355 } }
I20260925 04:43:20.373566 22392 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae07b292a70c4985bcf0f5c297af167c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:43:20.373646 22392 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae07b292a70c4985bcf0f5c297af167c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:43:20.373757 22392 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae07b292a70c4985bcf0f5c297af167c [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:43:20.374925 22392 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae07b292a70c4985bcf0f5c297af167c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae07b292a70c4985bcf0f5c297af167c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35355 } }
I20260925 04:43:20.375284 22392 leader_election.cc:302] T 00000000000000000000000000000000 P ae07b292a70c4985bcf0f5c297af167c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae07b292a70c4985bcf0f5c297af167c; no voters:
I20260925 04:43:20.375763 22392 leader_election.cc:288] T 00000000000000000000000000000000 P ae07b292a70c4985bcf0f5c297af167c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:43:20.375854 22397 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae07b292a70c4985bcf0f5c297af167c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:43:20.376034 22397 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae07b292a70c4985bcf0f5c297af167c [term 1 LEADER]: Becoming Leader. State: Replica: ae07b292a70c4985bcf0f5c297af167c, State: Running, Role: LEADER
I20260925 04:43:20.376408 22397 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae07b292a70c4985bcf0f5c297af167c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae07b292a70c4985bcf0f5c297af167c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35355 } }
I20260925 04:43:20.377180 22392 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae07b292a70c4985bcf0f5c297af167c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:43:20.378525 22398 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae07b292a70c4985bcf0f5c297af167c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae07b292a70c4985bcf0f5c297af167c. Latest consensus state: current_term: 1 leader_uuid: "ae07b292a70c4985bcf0f5c297af167c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae07b292a70c4985bcf0f5c297af167c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35355 } } }
I20260925 04:43:20.378551 22399 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae07b292a70c4985bcf0f5c297af167c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae07b292a70c4985bcf0f5c297af167c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae07b292a70c4985bcf0f5c297af167c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35355 } } }
I20260925 04:43:20.378746 22398 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae07b292a70c4985bcf0f5c297af167c [sys.catalog]: This master's current role is: LEADER
I20260925 04:43:20.378767 22399 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae07b292a70c4985bcf0f5c297af167c [sys.catalog]: This master's current role is: LEADER
I20260925 04:43:20.379071 22395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:20.364987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36467 (local address 127.0.98.62:35355)
0925 04:43:20.365460 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:20.365477 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:20.365518 (+ 41us) server_negotiation.cc:407] Connection header received
0925 04:43:20.366276 (+ 758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:20.366307 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:20.366680 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:20.366986 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:20.367911 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:20.368973 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:20.370166 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:20.370674 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:20.370802 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:20.373776 (+ 2974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:43:20.373782 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:43:20.373798 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:43:20.373844 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:43:20.375683 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:20.376205 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:20.376211 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:20.376218 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:20.376379 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:20.376663 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:20.376667 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:20.376669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:20.377136 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:43:20.377374 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:20.377585 (+ 211us) server_negotiation.cc:299] Negotiation successful
0925 04:43:20.377896 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":153,"threads_started":1}
I20260925 04:43:20.379163 22408 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:43:20.382427 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:43:20.382478 22408 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:43:20.390053 22408 catalog_manager.cc:1347] Generated new cluster ID: 935d2c303db248a1b35a9435acad8848
I20260925 04:43:20.390211 22408 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:43:20.392999 22395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:20.384353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52837 (local address 127.0.98.62:35355)
0925 04:43:20.384467 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:20.384471 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:20.384676 (+ 205us) server_negotiation.cc:407] Connection header received
0925 04:43:20.384860 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:20.384864 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:20.384911 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:20.384959 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:20.385902 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:20.386696 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:20.387493 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:20.387844 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:20.387962 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:20.389541 (+ 1579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:43:20.389545 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:43:20.389549 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:43:20.389577 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:43:20.391304 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:20.391795 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:20.391799 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:20.391801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:20.391904 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:20.392143 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:20.392146 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:20.392148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:20.392371 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:43:20.392499 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:20.392688 (+ 189us) server_negotiation.cc:299] Negotiation successful
0925 04:43:20.392815 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 04:43:20.407516 22408 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:43:20.408936 22408 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:43:20.414961 22408 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae07b292a70c4985bcf0f5c297af167c: Generated new TSK 0
I20260925 04:43:20.415900 22408 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:43:20.420431 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 22324
2026-09-25T04:43:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1341 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790311123478858-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:43:20 dist-test-slave-dc95 krb5kdc[22423](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:43:20 dist-test-slave-dc95 krb5kdc[22423](info): set up 2 sockets
Sep 25 04:43:20 dist-test-slave-dc95 krb5kdc[22423](info): commencing operation
krb5kdc: starting...
W20260925 04:43:21.379899 392 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790311123478858-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790311123478858-392-0/krb5kdc/test-user.keytab.
Sep 25 04:43:21 dist-test-slave-dc95 krb5kdc[22423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:43:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:43:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 04:43:21.507680 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790311123478858-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790311123478858-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790311123478858-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41811
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41811
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:43:21.618362 22436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:43:21.618664 22436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:43:21.618706 22436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:43:21.622872 22436 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:43:21.622963 22436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:43:21.622982 22436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:43:21.622998 22436 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:43:21.623013 22436 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:43:21.628799 22436 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790311123478858-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790311123478858-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41811
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790311123478858-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:43:21.629936 22436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:43:21.631024 22436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:43:21.639814 22441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:43:21.639815 22444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:43:21.639816 22442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:43:21.640005 22436 server_base.cc:1034] running on GCE node
I20260925 04:43:21.640787 22436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:43:21.641985 22436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:43:21.643185 22436 hybrid_clock.cc:648] HybridClock initialized: now 1790311401643170 us; error 35 us; skew 500 ppm
Sep 25 04:43:21 dist-test-slave-dc95 krb5kdc[22423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311401, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:43:21.646164 22436 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 04:43:21.647112 22436 webserver.cc:460] Webserver started at http://127.0.98.62:33657/ using document root <none> and password file <none>
I20260925 04:43:21.647704 22436 fs_manager.cc:359] Metadata directory not provided
I20260925 04:43:21.647812 22436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:43:21.648033 22436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:43:21.650144 22436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790311123478858-392-0/minicluster-data/master-0/data/instance:
uuid: "2794cf61c097486db9ab4fb37e8680f5"
format_stamp: "Formatted at 2026-09-25 04:43:21 on dist-test-slave-dc95"
server_key: "ba181996127f8b5386b3d5636f51e7d4"
server_key_iv: "abb270a6649ea422b673f774f292ac71"
server_key_version: "encryptionkey@0"
I20260925 04:43:21.650713 22436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790311123478858-392-0/minicluster-data/master-0/wal/instance:
uuid: "2794cf61c097486db9ab4fb37e8680f5"
format_stamp: "Formatted at 2026-09-25 04:43:21 on dist-test-slave-dc95"
server_key: "ba181996127f8b5386b3d5636f51e7d4"
server_key_iv: "abb270a6649ea422b673f774f292ac71"
server_key_version: "encryptionkey@0"
I20260925 04:43:21.654723 22436 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 04:43:21.657609 22451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:43:21.659049 22436 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:43:21.659214 22436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790311123478858-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790311123478858-392-0/minicluster-data/master-0/wal
uuid: "2794cf61c097486db9ab4fb37e8680f5"
format_stamp: "Formatted at 2026-09-25 04:43:21 on dist-test-slave-dc95"
server_key: "ba181996127f8b5386b3d5636f51e7d4"
server_key_iv: "abb270a6649ea422b673f774f292ac71"
server_key_version: "encryptionkey@0"
I20260925 04:43:21.659336 22436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790311123478858-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790311123478858-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790311123478858-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:43:21.671020 22436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:43:21.674301 22436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:43:21.674491 22436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:43:21.681643 22436 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41811
I20260925 04:43:21.681651 22503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41811 every 8 connection(s)
I20260925 04:43:21.682627 22436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
I20260925 04:43:21.684211 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 22436
I20260925 04:43:21.684333 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790311123478858-392-0/minicluster-data/master-0/wal/instance
I20260925 04:43:21.684625 392 external_mini_cluster.cc:1428] Setting key 903233bc3855a179ac99ff49457bcdfe
I20260925 04:43:21.686465 22504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:43:21.692121 22504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2794cf61c097486db9ab4fb37e8680f5: Bootstrap starting.
I20260925 04:43:21.693418 22507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:21.685860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35869 (local address 127.0.98.62:41811)
0925 04:43:21.686332 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:21.686348 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:21.686385 (+ 37us) server_negotiation.cc:407] Connection header received
0925 04:43:21.687081 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:21.687100 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:21.687470 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:21.687824 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:21.688234 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:21.689242 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:21.690100 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:21.690565 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:21.690715 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:21.691654 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:43:21.691662 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:43:21.691679 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:43:21.692105 (+ 426us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:43:21.692244 (+ 139us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:43:21.692277 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:43:21.692361 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:21.692492 (+ 131us) server_negotiation.cc:299] Negotiation successful
0925 04:43:21.692706 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":120,"threads_started":1}
I20260925 04:43:21.694476 22504 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2794cf61c097486db9ab4fb37e8680f5: Neither blocks nor log segments found. Creating new log.
I20260925 04:43:21.695326 22504 log.cc:824] T 00000000000000000000000000000000 P 2794cf61c097486db9ab4fb37e8680f5: Log is configured to *not* fsync() on all Append() calls
I20260925 04:43:21.696137 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:43:21.697227 22504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2794cf61c097486db9ab4fb37e8680f5: No bootstrap required, opened a new log
I20260925 04:43:21.701689 22504 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2794cf61c097486db9ab4fb37e8680f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2794cf61c097486db9ab4fb37e8680f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41811 } }
I20260925 04:43:21.702003 22507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:21.698016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57403 (local address 127.0.98.62:41811)
0925 04:43:21.698186 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:21.698190 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:21.698276 (+ 86us) server_negotiation.cc:407] Connection header received
0925 04:43:21.698397 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:21.698400 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:21.698436 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:21.698494 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:21.698739 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:21.699306 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:21.700487 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:21.700721 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:21.700806 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:21.701565 (+ 759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:43:21.701568 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:43:21.701570 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:43:21.701582 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:43:21.701605 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:43:21.701615 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:43:21.701669 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:21.701831 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 04:43:21.701889 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260925 04:43:21.702020 22504 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2794cf61c097486db9ab4fb37e8680f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:43:21.702289 22504 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2794cf61c097486db9ab4fb37e8680f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2794cf61c097486db9ab4fb37e8680f5, State: Initialized, Role: FOLLOWER
I20260925 04:43:21.702739 22504 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2794cf61c097486db9ab4fb37e8680f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2794cf61c097486db9ab4fb37e8680f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41811 } }
I20260925 04:43:21.702869 22504 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2794cf61c097486db9ab4fb37e8680f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:43:21.702939 22504 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2794cf61c097486db9ab4fb37e8680f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:43:21.703044 22504 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2794cf61c097486db9ab4fb37e8680f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:43:21.704057 22504 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2794cf61c097486db9ab4fb37e8680f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2794cf61c097486db9ab4fb37e8680f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41811 } }
I20260925 04:43:21.704408 22504 leader_election.cc:302] T 00000000000000000000000000000000 P 2794cf61c097486db9ab4fb37e8680f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2794cf61c097486db9ab4fb37e8680f5; no voters:
I20260925 04:43:21.704715 22504 leader_election.cc:288] T 00000000000000000000000000000000 P 2794cf61c097486db9ab4fb37e8680f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:43:21.704877 22514 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2794cf61c097486db9ab4fb37e8680f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:43:21.705090 22514 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2794cf61c097486db9ab4fb37e8680f5 [term 1 LEADER]: Becoming Leader. State: Replica: 2794cf61c097486db9ab4fb37e8680f5, State: Running, Role: LEADER
I20260925 04:43:21.705418 22514 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2794cf61c097486db9ab4fb37e8680f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2794cf61c097486db9ab4fb37e8680f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41811 } }
I20260925 04:43:21.706000 22504 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2794cf61c097486db9ab4fb37e8680f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:43:21.707182 22515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2794cf61c097486db9ab4fb37e8680f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2794cf61c097486db9ab4fb37e8680f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2794cf61c097486db9ab4fb37e8680f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41811 } } }
I20260925 04:43:21.707173 22516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2794cf61c097486db9ab4fb37e8680f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2794cf61c097486db9ab4fb37e8680f5. Latest consensus state: current_term: 1 leader_uuid: "2794cf61c097486db9ab4fb37e8680f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2794cf61c097486db9ab4fb37e8680f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41811 } } }
I20260925 04:43:21.707322 22515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2794cf61c097486db9ab4fb37e8680f5 [sys.catalog]: This master's current role is: LEADER
I20260925 04:43:21.707356 22516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2794cf61c097486db9ab4fb37e8680f5 [sys.catalog]: This master's current role is: LEADER
I20260925 04:43:21.707990 22523 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:43:21.711061 22523 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:43:21.716866 22523 catalog_manager.cc:1347] Generated new cluster ID: b8412f5d778145eba4a08effa8801b78
I20260925 04:43:21.716955 22523 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:43:21.727844 22523 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:43:21.729180 22523 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:43:21.734433 22523 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2794cf61c097486db9ab4fb37e8680f5: Generated new TSK 0
I20260925 04:43:21.735289 22523 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:43:21.746524 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 22436
2026-09-25T04:43:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1325 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790311123478858-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:43:21 dist-test-slave-dc95 krb5kdc[22535](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:43:21 dist-test-slave-dc95 krb5kdc[22535](info): set up 2 sockets
Sep 25 04:43:21 dist-test-slave-dc95 krb5kdc[22535](info): commencing operation
krb5kdc: starting...
W20260925 04:43:22.715831 392 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790311123478858-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790311123478858-392-0/krb5kdc/test-user.keytab.
Sep 25 04:43:22 dist-test-slave-dc95 krb5kdc[22535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:43:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:43:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 04:43:22.838292 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790311123478858-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790311123478858-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790311123478858-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39985
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39985
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:43:22.951367 22548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:43:22.951702 22548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:43:22.951746 22548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:43:22.956126 22548 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:43:22.956199 22548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:43:22.956218 22548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:43:22.956233 22548 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:43:22.956248 22548 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:43:22.961788 22548 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790311123478858-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790311123478858-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39985
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790311123478858-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:43:22.962862 22548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:43:22.963956 22548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:43:22.972841 22553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:43:22.972841 22556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:43:22.972860 22554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:43:22.973162 22548 server_base.cc:1034] running on GCE node
I20260925 04:43:22.973731 22548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:43:22.974941 22548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:43:22.976114 22548 hybrid_clock.cc:648] HybridClock initialized: now 1790311402976089 us; error 36 us; skew 500 ppm
Sep 25 04:43:22 dist-test-slave-dc95 krb5kdc[22535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:43:22.978901 22548 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 04:43:22.979882 22548 webserver.cc:460] Webserver started at http://127.0.98.62:36341/ using document root <none> and password file <none>
I20260925 04:43:22.980429 22548 fs_manager.cc:359] Metadata directory not provided
I20260925 04:43:22.980494 22548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:43:22.980679 22548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:43:22.982753 22548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790311123478858-392-0/minicluster-data/master-0/data/instance:
uuid: "b5852b0504b04ad3896bb422b63495e2"
format_stamp: "Formatted at 2026-09-25 04:43:22 on dist-test-slave-dc95"
server_key: "471528f070189c039237bc57c7c7a724"
server_key_iv: "c91a54c9a903bfec2468dc352fff09b2"
server_key_version: "encryptionkey@0"
I20260925 04:43:22.983299 22548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790311123478858-392-0/minicluster-data/master-0/wal/instance:
uuid: "b5852b0504b04ad3896bb422b63495e2"
format_stamp: "Formatted at 2026-09-25 04:43:22 on dist-test-slave-dc95"
server_key: "471528f070189c039237bc57c7c7a724"
server_key_iv: "c91a54c9a903bfec2468dc352fff09b2"
server_key_version: "encryptionkey@0"
I20260925 04:43:22.987082 22548 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 04:43:22.989634 22563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:43:22.990943 22548 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:43:22.991103 22548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790311123478858-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790311123478858-392-0/minicluster-data/master-0/wal
uuid: "b5852b0504b04ad3896bb422b63495e2"
format_stamp: "Formatted at 2026-09-25 04:43:22 on dist-test-slave-dc95"
server_key: "471528f070189c039237bc57c7c7a724"
server_key_iv: "c91a54c9a903bfec2468dc352fff09b2"
server_key_version: "encryptionkey@0"
I20260925 04:43:22.991215 22548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790311123478858-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790311123478858-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790311123478858-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:43:22.996042 22548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:43:22.999054 22548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:43:22.999217 22548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:43:23.005671 22548 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39985
I20260925 04:43:23.005715 22615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39985 every 8 connection(s)
I20260925 04:43:23.006599 22548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
I20260925 04:43:23.009673 22616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:43:23.014629 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 22548
I20260925 04:43:23.014768 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790311123478858-392-0/minicluster-data/master-0/wal/instance
I20260925 04:43:23.015019 392 external_mini_cluster.cc:1428] Setting key 6d3f02da5a32b629b81d967deded8d0e
I20260925 04:43:23.015400 22616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5852b0504b04ad3896bb422b63495e2: Bootstrap starting.
I20260925 04:43:23.018121 22616 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5852b0504b04ad3896bb422b63495e2: Neither blocks nor log segments found. Creating new log.
I20260925 04:43:23.019064 22616 log.cc:824] T 00000000000000000000000000000000 P b5852b0504b04ad3896bb422b63495e2: Log is configured to *not* fsync() on all Append() calls
I20260925 04:43:23.020743 22619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:23.016495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36175 (local address 127.0.98.62:39985)
0925 04:43:23.016978 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:23.017001 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:23.017044 (+ 43us) server_negotiation.cc:407] Connection header received
0925 04:43:23.017914 (+ 870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:23.017940 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:23.018345 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:23.018766 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:23.019459 (+ 693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:43:23.019465 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:43:23.019481 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:43:23.019494 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:43:23.019557 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:43:23.019580 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:43:23.019702 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:23.019829 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 04:43:23.019979 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":144,"threads_started":1}
I20260925 04:43:23.021240 22616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5852b0504b04ad3896bb422b63495e2: No bootstrap required, opened a new log
I20260925 04:43:23.023545 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:43:23.025483 22616 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5852b0504b04ad3896bb422b63495e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5852b0504b04ad3896bb422b63495e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39985 } }
I20260925 04:43:23.025792 22616 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5852b0504b04ad3896bb422b63495e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:43:23.025873 22616 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5852b0504b04ad3896bb422b63495e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5852b0504b04ad3896bb422b63495e2, State: Initialized, Role: FOLLOWER
I20260925 04:43:23.026378 22616 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5852b0504b04ad3896bb422b63495e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5852b0504b04ad3896bb422b63495e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39985 } }
I20260925 04:43:23.026530 22616 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5852b0504b04ad3896bb422b63495e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:43:23.026597 22616 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5852b0504b04ad3896bb422b63495e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:43:23.026703 22616 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5852b0504b04ad3896bb422b63495e2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:43:23.027837 22619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:23.025970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50603 (local address 127.0.98.62:39985)
0925 04:43:23.026110 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:23.026115 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:23.026500 (+ 385us) server_negotiation.cc:407] Connection header received
0925 04:43:23.026626 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:23.026629 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:23.026665 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:23.026738 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:23.027425 (+ 687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:43:23.027428 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:43:23.027430 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:43:23.027441 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:43:23.027467 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:43:23.027476 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:43:23.027550 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:23.027679 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 04:43:23.027716 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 04:43:23.027982 22616 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5852b0504b04ad3896bb422b63495e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5852b0504b04ad3896bb422b63495e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39985 } }
I20260925 04:43:23.028347 22616 leader_election.cc:302] T 00000000000000000000000000000000 P b5852b0504b04ad3896bb422b63495e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5852b0504b04ad3896bb422b63495e2; no voters:
I20260925 04:43:23.028661 22616 leader_election.cc:288] T 00000000000000000000000000000000 P b5852b0504b04ad3896bb422b63495e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:43:23.028743 22626 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5852b0504b04ad3896bb422b63495e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:43:23.028955 22626 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5852b0504b04ad3896bb422b63495e2 [term 1 LEADER]: Becoming Leader. State: Replica: b5852b0504b04ad3896bb422b63495e2, State: Running, Role: LEADER
I20260925 04:43:23.029364 22626 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5852b0504b04ad3896bb422b63495e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5852b0504b04ad3896bb422b63495e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39985 } }
I20260925 04:43:23.029868 22616 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5852b0504b04ad3896bb422b63495e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:43:23.030867 22627 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5852b0504b04ad3896bb422b63495e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5852b0504b04ad3896bb422b63495e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5852b0504b04ad3896bb422b63495e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39985 } } }
I20260925 04:43:23.030992 22627 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5852b0504b04ad3896bb422b63495e2 [sys.catalog]: This master's current role is: LEADER
I20260925 04:43:23.031332 22628 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5852b0504b04ad3896bb422b63495e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5852b0504b04ad3896bb422b63495e2. Latest consensus state: current_term: 1 leader_uuid: "b5852b0504b04ad3896bb422b63495e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5852b0504b04ad3896bb422b63495e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39985 } } }
I20260925 04:43:23.031384 22635 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:43:23.031483 22628 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5852b0504b04ad3896bb422b63495e2 [sys.catalog]: This master's current role is: LEADER
I20260925 04:43:23.034343 22635 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:43:23.040409 22635 catalog_manager.cc:1347] Generated new cluster ID: 7948fde2c42b40ce8d6e205d575a6537
I20260925 04:43:23.040537 22635 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:43:23.064742 22635 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:43:23.065099 22635 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:43:23.078991 22635 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5852b0504b04ad3896bb422b63495e2: Generated new TSK 0
I20260925 04:43:23.079770 22635 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:43:23.084965 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 22548
2026-09-25T04:43:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1338 ms)
[----------] 18 tests from AuthTokenIssuingTest (24194 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:43:23 dist-test-slave-dc95 krb5kdc[22647](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:43:23 dist-test-slave-dc95 krb5kdc[22647](info): set up 2 sockets
Sep 25 04:43:23 dist-test-slave-dc95 krb5kdc[22647](info): commencing operation
krb5kdc: starting...
W20260925 04:43:24.051759 392 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/krb5kdc/test-user.keytab.
Sep 25 04:43:24 dist-test-slave-dc95 krb5kdc[22647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:43:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:43:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 04:43:24.174961 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38567
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38567,127.0.98.61:32855,127.0.98.60:41675
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:43:24.289899 22660 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 04:43:24.290230 22660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:43:24.290264 22660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:43:24.290323 22660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:43:24.294612 22660 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:43:24.294693 22660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:43:24.294721 22660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:43:24.294744 22660 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:43:24.294765 22660 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:43:24.300418 22660 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38567,127.0.98.61:32855,127.0.98.60:41675
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22660
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:43:24.301528 22660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:43:24.302543 22660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:43:24.311264 22665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:43:24.311403 22668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:43:24.311302 22666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:43:24.311362 22660 server_base.cc:1034] running on GCE node
I20260925 04:43:24.312151 22660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:43:24.313360 22660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:43:24.314586 22660 hybrid_clock.cc:648] HybridClock initialized: now 1790311404314566 us; error 37 us; skew 500 ppm
Sep 25 04:43:24 dist-test-slave-dc95 krb5kdc[22647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:43:24.317723 22660 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 04:43:24.318809 22660 webserver.cc:460] Webserver started at http://127.0.98.62:36567/ using document root <none> and password file <none>
I20260925 04:43:24.319378 22660 fs_manager.cc:359] Metadata directory not provided
I20260925 04:43:24.319451 22660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:43:24.319705 22660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:43:24.322348 22660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/minicluster-data/master-0/data/instance:
uuid: "91f939ff44804b7b9633ff3df1c9aafe"
format_stamp: "Formatted at 2026-09-25 04:43:24 on dist-test-slave-dc95"
server_key: "e387e830eb301a102cee504c6ca50e08"
server_key_iv: "61fdad219d833f3bccf6a75029490096"
server_key_version: "encryptionkey@0"
I20260925 04:43:24.323096 22660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/minicluster-data/master-0/wal/instance:
uuid: "91f939ff44804b7b9633ff3df1c9aafe"
format_stamp: "Formatted at 2026-09-25 04:43:24 on dist-test-slave-dc95"
server_key: "e387e830eb301a102cee504c6ca50e08"
server_key_iv: "61fdad219d833f3bccf6a75029490096"
server_key_version: "encryptionkey@0"
I20260925 04:43:24.327845 22660 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260925 04:43:24.330484 22675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:43:24.332019 22660 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:43:24.332163 22660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/minicluster-data/master-0/wal
uuid: "91f939ff44804b7b9633ff3df1c9aafe"
format_stamp: "Formatted at 2026-09-25 04:43:24 on dist-test-slave-dc95"
server_key: "e387e830eb301a102cee504c6ca50e08"
server_key_iv: "61fdad219d833f3bccf6a75029490096"
server_key_version: "encryptionkey@0"
I20260925 04:43:24.332278 22660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:43:24.348271 22660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:43:24.351434 22660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:43:24.351675 22660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:43:24.358595 22660 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38567
I20260925 04:43:24.358913 22727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38567 every 8 connection(s)
I20260925 04:43:24.359579 22660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
I20260925 04:43:24.362047 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 22660
I20260925 04:43:24.362190 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/minicluster-data/master-0/wal/instance
I20260925 04:43:24.362481 392 external_mini_cluster.cc:1428] Setting key c9adc21ac11a303a06c47a66468f2422
I20260925 04:43:24.362651 22728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:43:24.366899 22728 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38567 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 25 04:43:24 dist-test-slave-dc95 krb5kdc[22647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:43:24.383533 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:24.369302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38567 (local address 127.0.0.1:39816)
0925 04:43:24.369906 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0925 04:43:24.369956 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 04:43:24.370875 (+ 919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:43:24.371740 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:43:24.371746 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:43:24.372105 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:43:24.372394 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:43:24.372420 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:43:24.373692 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:43:24.373695 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:43:24.374632 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:43:24.374642 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:43:24.375530 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:43:24.375533 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:43:24.375665 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:24.376264 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:43:24.376281 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:43:24.378122 (+ 1841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:43:24.380271 (+ 2149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:43:24.380282 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:43:24.380295 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:43:24.380617 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:43:24.380998 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:43:24.381001 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:43:24.381002 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:43:24.381129 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:43:24.381967 (+ 838us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:43:24.381973 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:43:24.382107 (+ 134us) client_negotiation.cc:769] Sending connection context
0925 04:43:24.382268 (+ 161us) client_negotiation.cc:240] Negotiation successful
0925 04:43:24.382711 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"mutex_wait_us":109,"thread_start_us":132,"threads_started":1}
I20260925 04:43:24.383538 22732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:24.369302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39816 (local address 127.0.98.62:38567)
0925 04:43:24.369842 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:24.369859 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:24.370092 (+ 233us) server_negotiation.cc:407] Connection header received
0925 04:43:24.371193 (+ 1101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:24.371198 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:24.371407 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:24.371591 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:24.372583 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:24.373551 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:24.374767 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:24.375344 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:24.375520 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:24.378317 (+ 2797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:43:24.378323 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:43:24.378335 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:43:24.378359 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:43:24.380141 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:24.380745 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:24.380750 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:24.380752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:24.380869 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:24.381247 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:24.381285 (+ 38us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:24.381286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:24.381781 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:43:24.381921 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:24.382358 (+ 437us) server_negotiation.cc:299] Negotiation successful
0925 04:43:24.382564 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"spinlock_wait_cycles":54656,"thread_start_us":152,"threads_started":1}
I20260925 04:43:24.386046 22728 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32855 } has no permanent_uuid. Determining permanent_uuid...
I20260925 04:43:24.386817 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:24.386548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32855 (local address 127.0.0.1:38942)
0925 04:43:24.386674 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 04:43:24.386748 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 04:43:24.387013 22676 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32855: connect: Connection refused (error 111)
I20260925 04:43:24.387871 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:24.387666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32855 (local address 127.0.0.1:38952)
0925 04:43:24.387769 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0925 04:43:24.387814 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 04:43:24.388039 22728 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:32855: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32855: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260925 04:43:24.390740 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:32855
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38567,127.0.98.61:32855,127.0.98.60:41675
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:43:24.437249 22728 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32855 } attempt: 1
I20260925 04:43:24.438170 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:24.437859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32855 (local address 127.0.0.1:38960)
0925 04:43:24.438020 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 04:43:24.438088 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260925 04:43:24.438911 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:24.438691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32855 (local address 127.0.0.1:38976)
0925 04:43:24.438800 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0925 04:43:24.438846 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260925 04:43:24.439064 22728 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:32855: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32855: connect: Connection refused (error 111)
W20260925 04:43:24.505514 22735 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 04:43:24.505857 22735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:43:24.505895 22735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:43:24.505959 22735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 04:43:24.507223 22728 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32855 } attempt: 2
I20260925 04:43:24.508160 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:24.507841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32855 (local address 127.0.0.1:38984)
0925 04:43:24.508004 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 04:43:24.508070 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 04:43:24.508893 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:24.508673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32855 (local address 127.0.0.1:38990)
0925 04:43:24.508780 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0925 04:43:24.508830 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 04:43:24.509042 22728 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:32855: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32855: connect: Connection refused (error 111)
W20260925 04:43:24.510596 22735 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:43:24.510660 22735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:43:24.510689 22735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:43:24.510725 22735 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:43:24.510754 22735 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:43:24.516503 22735 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38567,127.0.98.61:32855,127.0.98.60:41675
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:32855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:43:24.517668 22735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:43:24.518736 22735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 04:43:24.527738 22735 server_base.cc:1034] running on GCE node
W20260925 04:43:24.527685 22740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:43:24.527694 22741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:43:24.527685 22743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:43:24.528612 22735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:43:24.529847 22735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:43:24.531028 22735 hybrid_clock.cc:648] HybridClock initialized: now 1790311404531001 us; error 45 us; skew 500 ppm
Sep 25 04:43:24 dist-test-slave-dc95 krb5kdc[22647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:43:24.534003 22735 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260925 04:43:24.534946 22735 webserver.cc:460] Webserver started at http://127.0.98.61:38421/ using document root <none> and password file <none>
I20260925 04:43:24.535517 22735 fs_manager.cc:359] Metadata directory not provided
I20260925 04:43:24.535580 22735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:43:24.535831 22735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:43:24.538044 22735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/minicluster-data/master-1/data/instance:
uuid: "aa063903041144afa6ecc2495bed0c60"
format_stamp: "Formatted at 2026-09-25 04:43:24 on dist-test-slave-dc95"
server_key: "d1914fb9a9d3e4a37f33d72105995549"
server_key_iv: "3855978c3c501eb33fbcde08b76d3ed2"
server_key_version: "encryptionkey@0"
I20260925 04:43:24.538580 22735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/minicluster-data/master-1/wal/instance:
uuid: "aa063903041144afa6ecc2495bed0c60"
format_stamp: "Formatted at 2026-09-25 04:43:24 on dist-test-slave-dc95"
server_key: "d1914fb9a9d3e4a37f33d72105995549"
server_key_iv: "3855978c3c501eb33fbcde08b76d3ed2"
server_key_version: "encryptionkey@0"
I20260925 04:43:24.542520 22735 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 04:43:24.545255 22750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:43:24.546586 22735 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 04:43:24.546702 22735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/minicluster-data/master-1/wal
uuid: "aa063903041144afa6ecc2495bed0c60"
format_stamp: "Formatted at 2026-09-25 04:43:24 on dist-test-slave-dc95"
server_key: "d1914fb9a9d3e4a37f33d72105995549"
server_key_iv: "3855978c3c501eb33fbcde08b76d3ed2"
server_key_version: "encryptionkey@0"
I20260925 04:43:24.546831 22735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:43:24.560603 22735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:43:24.563870 22735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:43:24.564041 22735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:43:24.571066 22735 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:32855
I20260925 04:43:24.571362 22802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:32855 every 8 connection(s)
I20260925 04:43:24.572134 22735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/minicluster-data/master-1/data/info.pb
I20260925 04:43:24.575026 22803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:43:24.577314 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 22735
I20260925 04:43:24.577425 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/minicluster-data/master-1/wal/instance
I20260925 04:43:24.577700 392 external_mini_cluster.cc:1428] Setting key fbbb659383f9ce895519fd0b2fb37f63
I20260925 04:43:24.579413 22803 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38567 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 25 04:43:24 dist-test-slave-dc95 krb5kdc[22647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:43:24.595455 22732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:24.582035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39830 (local address 127.0.98.62:38567)
0925 04:43:24.582213 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:24.582218 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:24.583319 (+ 1101us) server_negotiation.cc:407] Connection header received
0925 04:43:24.584717 (+ 1398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:24.584720 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:24.584777 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:24.584864 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:24.586619 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:24.587178 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:24.588483 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:24.588743 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:24.588853 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:24.591820 (+ 2967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:43:24.591823 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:43:24.591825 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:43:24.591857 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:43:24.593221 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:24.593805 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:24.593808 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:24.593809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:24.593893 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:24.594248 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:24.594251 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:24.594252 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:24.594460 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:43:24.594549 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:24.595131 (+ 582us) server_negotiation.cc:299] Negotiation successful
0925 04:43:24.595268 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 04:43:24.596395 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:24.582504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38567 (local address 127.0.0.1:39830)
0925 04:43:24.583099 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0925 04:43:24.583169 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0925 04:43:24.584429 (+ 1260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:43:24.585115 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:43:24.585126 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:43:24.585630 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:43:24.586401 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:43:24.586427 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:43:24.587374 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:43:24.587377 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:43:24.588347 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:43:24.588359 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:43:24.588956 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:43:24.588961 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:43:24.589197 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:24.589912 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:43:24.589930 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:43:24.591593 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:43:24.593359 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:43:24.593366 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:43:24.593385 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:43:24.593679 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:43:24.593982 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:43:24.593985 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:43:24.593987 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:43:24.594146 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:43:24.594564 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:43:24.594571 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:43:24.594864 (+ 293us) client_negotiation.cc:769] Sending connection context
0925 04:43:24.595037 (+ 173us) client_negotiation.cc:240] Negotiation successful
0925 04:43:24.595400 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":130,"threads_started":1}
I20260925 04:43:24.597873 22803 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32855 } has no permanent_uuid. Determining permanent_uuid...
I20260925 04:43:24.600189 22728 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32855 } attempt: 3
Sep 25 04:43:24 dist-test-slave-dc95 krb5kdc[22647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 25 04:43:24 dist-test-slave-dc95 krb5kdc[22647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260925 04:43:24.607161 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41675
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38567,127.0.98.61:32855,127.0.98.60:41675
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:43:24.607820 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:24.598379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32855 (local address 127.0.0.1:39002)
0925 04:43:24.598491 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0925 04:43:24.598504 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:43:24.598578 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:43:24.599118 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:43:24.599121 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:43:24.599304 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:43:24.599391 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:43:24.599397 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:43:24.600384 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:43:24.600388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:43:24.601112 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:43:24.601123 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:43:24.601874 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:43:24.601879 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:43:24.602026 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:24.602588 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:43:24.602599 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:43:24.604023 (+ 1424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:43:24.606069 (+ 2046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:43:24.606072 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:43:24.606074 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:43:24.606299 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:43:24.606577 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:43:24.606580 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:43:24.606582 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:43:24.606665 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:43:24.607213 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:43:24.607216 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:43:24.607325 (+ 109us) client_negotiation.cc:769] Sending connection context
0925 04:43:24.607440 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 04:43:24.607595 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 04:43:24.608112 22808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:24.598485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39002 (local address 127.0.98.61:32855)
0925 04:43:24.598786 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:24.598794 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:24.598812 (+ 18us) server_negotiation.cc:407] Connection header received
0925 04:43:24.598844 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:24.598848 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:24.599004 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:24.599123 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:24.599493 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:24.600283 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:24.601233 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:24.601776 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:24.601856 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:24.604168 (+ 2312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:43:24.604179 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:43:24.604193 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:43:24.604215 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:43:24.605970 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:24.606411 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:24.606417 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:24.606418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:24.606493 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:24.606758 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:24.606761 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:24.606763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:24.607107 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:43:24.607243 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:24.607782 (+ 539us) server_negotiation.cc:299] Negotiation successful
0925 04:43:24.607943 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":116,"threads_started":1}
I20260925 04:43:24.609941 22803 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41675 } has no permanent_uuid. Determining permanent_uuid...
I20260925 04:43:24.610658 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:24.610375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41675 (local address 127.0.0.1:42066)
0925 04:43:24.610513 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 04:43:24.610594 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260925 04:43:24.610872 22752 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41675: connect: Connection refused (error 111)
I20260925 04:43:24.611333 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:24.600639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32855 (local address 127.0.0.1:39018)
0925 04:43:24.600755 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 04:43:24.600770 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:43:24.600838 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:43:24.601267 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:43:24.601270 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:43:24.601455 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:43:24.601568 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:43:24.601575 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:43:24.602589 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:43:24.602593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:43:24.603895 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:43:24.603908 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:43:24.604474 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:43:24.604477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:43:24.604586 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:24.604967 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:43:24.604978 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:43:24.606321 (+ 1343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:43:24.609341 (+ 3020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:43:24.609344 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:43:24.609346 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:43:24.609602 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:43:24.610290 (+ 688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:43:24.610293 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:43:24.610295 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:43:24.610401 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:43:24.610839 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:43:24.610842 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:43:24.610947 (+ 105us) client_negotiation.cc:769] Sending connection context
0925 04:43:24.611024 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 04:43:24.611182 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260925 04:43:24.611837 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:24.611640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41675 (local address 127.0.0.1:42074)
0925 04:43:24.611727 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0925 04:43:24.611776 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260925 04:43:24.611986 22803 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41675: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41675: connect: Connection refused (error 111)
I20260925 04:43:24.612147 22809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:24.600725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39018 (local address 127.0.98.61:32855)
0925 04:43:24.601053 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:24.601059 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:24.601070 (+ 11us) server_negotiation.cc:407] Connection header received
0925 04:43:24.601096 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:24.601098 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:24.601154 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:24.601236 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:24.601686 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:24.602460 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:24.604013 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:24.604342 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:24.604456 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:24.606468 (+ 2012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:43:24.606472 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:43:24.606473 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:43:24.606507 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:43:24.609218 (+ 2711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:24.609703 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:24.609707 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:24.609708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:24.609795 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:24.610505 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:24.610508 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:24.610509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:24.610715 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:43:24.611724 (+ 1009us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:24.611815 (+ 91us) server_negotiation.cc:299] Negotiation successful
0925 04:43:24.611943 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":272,"spinlock_wait_cycles":1920,"thread_start_us":162,"threads_started":1}
I20260925 04:43:24.612820 22728 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41675 } has no permanent_uuid. Determining permanent_uuid...
I20260925 04:43:24.613547 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:24.613276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41675 (local address 127.0.0.1:42090)
0925 04:43:24.613423 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 04:43:24.613474 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260925 04:43:24.614238 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:24.614007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41675 (local address 127.0.0.1:42096)
0925 04:43:24.614143 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 04:43:24.614185 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260925 04:43:24.614446 22728 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41675: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41675: connect: Connection refused (error 111)
I20260925 04:43:24.645648 22728 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41675 } attempt: 1
I20260925 04:43:24.646549 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:24.646225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41675 (local address 127.0.0.1:42108)
0925 04:43:24.646394 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 04:43:24.646460 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260925 04:43:24.647307 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:24.647063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41675 (local address 127.0.0.1:42124)
0925 04:43:24.647178 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0925 04:43:24.647224 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 04:43:24.647429 22728 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41675: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41675: connect: Connection refused (error 111)
I20260925 04:43:24.661302 22803 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41675 } attempt: 1
I20260925 04:43:24.662173 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:24.661883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41675 (local address 127.0.0.1:42138)
0925 04:43:24.662028 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 04:43:24.662092 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260925 04:43:24.662935 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:24.662681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41675 (local address 127.0.0.1:42146)
0925 04:43:24.662836 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 04:43:24.662880 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
W20260925 04:43:24.663070 22803 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41675: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41675: connect: Connection refused (error 111)
I20260925 04:43:24.722605 22728 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41675 } attempt: 2
W20260925 04:43:24.722581 22810 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 04:43:24.722947 22810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:43:24.722986 22810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:43:24.723037 22810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 04:43:24.723524 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:24.723165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41675 (local address 127.0.0.1:42154)
0925 04:43:24.723339 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 04:43:24.723422 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260925 04:43:24.724249 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:24.724061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41675 (local address 127.0.0.1:42164)
0925 04:43:24.724153 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0925 04:43:24.724196 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 04:43:24.724375 22728 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41675: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41675: connect: Connection refused (error 111)
W20260925 04:43:24.727643 22810 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:43:24.727721 22810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:43:24.727751 22810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:43:24.727782 22810 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:43:24.727811 22810 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:43:24.731276 22803 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41675 } attempt: 2
I20260925 04:43:24.732219 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:24.731928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41675 (local address 127.0.0.1:42170)
0925 04:43:24.732064 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 04:43:24.732121 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260925 04:43:24.732900 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:24.732667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41675 (local address 127.0.0.1:42174)
0925 04:43:24.732807 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 04:43:24.732850 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260925 04:43:24.732996 22803 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41675: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41675: connect: Connection refused (error 111)
I20260925 04:43:24.733634 22810 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38567,127.0.98.61:32855,127.0.98.60:41675
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:43:24.734793 22810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:43:24.735843 22810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:43:24.744554 22816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:43:24.744575 22817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:43:24.744522 22819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:43:24.744966 22810 server_base.cc:1034] running on GCE node
I20260925 04:43:24.745535 22810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:43:24.746721 22810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:43:24.747918 22810 hybrid_clock.cc:648] HybridClock initialized: now 1790311404747895 us; error 41 us; skew 500 ppm
Sep 25 04:43:24 dist-test-slave-dc95 krb5kdc[22647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:43:24.750952 22810 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260925 04:43:24.752032 22810 webserver.cc:460] Webserver started at http://127.0.98.60:42705/ using document root <none> and password file <none>
I20260925 04:43:24.752588 22810 fs_manager.cc:359] Metadata directory not provided
I20260925 04:43:24.752662 22810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:43:24.752867 22810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:43:24.755029 22810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/minicluster-data/master-2/data/instance:
uuid: "3090682f59564033a4cd064e53f2d85d"
format_stamp: "Formatted at 2026-09-25 04:43:24 on dist-test-slave-dc95"
server_key: "fee31c21780d12273d5d8f93eed971af"
server_key_iv: "698a5317486e0002e1f71f1dc0c26cec"
server_key_version: "encryptionkey@0"
I20260925 04:43:24.755582 22810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/minicluster-data/master-2/wal/instance:
uuid: "3090682f59564033a4cd064e53f2d85d"
format_stamp: "Formatted at 2026-09-25 04:43:24 on dist-test-slave-dc95"
server_key: "fee31c21780d12273d5d8f93eed971af"
server_key_iv: "698a5317486e0002e1f71f1dc0c26cec"
server_key_version: "encryptionkey@0"
I20260925 04:43:24.759390 22810 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 04:43:24.762029 22826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:43:24.763315 22810 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:43:24.763444 22810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/minicluster-data/master-2/wal
uuid: "3090682f59564033a4cd064e53f2d85d"
format_stamp: "Formatted at 2026-09-25 04:43:24 on dist-test-slave-dc95"
server_key: "fee31c21780d12273d5d8f93eed971af"
server_key_iv: "698a5317486e0002e1f71f1dc0c26cec"
server_key_version: "encryptionkey@0"
I20260925 04:43:24.763549 22810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:43:24.794943 22810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:43:24.798432 22810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:43:24.798635 22810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:43:24.805698 22810 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:41675
I20260925 04:43:24.805953 22878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:41675 every 8 connection(s)
I20260925 04:43:24.806677 22810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/minicluster-data/master-2/data/info.pb
I20260925 04:43:24.809589 22879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:43:24.813810 22879 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38567 } has no permanent_uuid. Determining permanent_uuid...
I20260925 04:43:24.815788 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 22810
I20260925 04:43:24.815917 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311123478858-392-0/minicluster-data/master-2/wal/instance
I20260925 04:43:24.816179 392 external_mini_cluster.cc:1428] Setting key d4c9360b5227380d1777a5b9c4f35b85
Sep 25 04:43:24 dist-test-slave-dc95 krb5kdc[22647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 25 04:43:24 dist-test-slave-dc95 krb5kdc[22647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 04:43:24.823529 22728 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41675 } attempt: 3
I20260925 04:43:24.824133 22803 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41675 } attempt: 3
I20260925 04:43:24.828037 22732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:24.815889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39846 (local address 127.0.98.62:38567)
0925 04:43:24.816037 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:24.816042 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:24.817093 (+ 1051us) server_negotiation.cc:407] Connection header received
0925 04:43:24.818169 (+ 1076us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:24.818173 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:24.818238 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:24.818304 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:24.819188 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:24.819744 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:24.820701 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:24.820927 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:24.821015 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:24.823693 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:43:24.823698 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:43:24.823701 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:43:24.823733 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:43:24.825536 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:24.826167 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:24.826171 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:24.826172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:24.826260 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:24.826669 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:24.826684 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:24.826686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:24.826914 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:43:24.826983 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:24.827745 (+ 762us) server_negotiation.cc:299] Negotiation successful
0925 04:43:24.827870 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Sep 25 04:43:24 dist-test-slave-dc95 krb5kdc[22647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 04:43:24.829825 22881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:24.816393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38567 (local address 127.0.0.1:39846)
0925 04:43:24.816937 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0925 04:43:24.816991 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 04:43:24.817950 (+ 959us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:43:24.818371 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:43:24.818378 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:43:24.818741 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:43:24.819030 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:43:24.819043 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:43:24.819847 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:43:24.819850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:43:24.820581 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:43:24.820589 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:43:24.821065 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:43:24.821069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:43:24.821278 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:24.821841 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:43:24.821862 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:43:24.823445 (+ 1583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:43:24.825637 (+ 2192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:43:24.825647 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:43:24.825658 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:43:24.826061 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:43:24.826371 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:43:24.826374 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:43:24.826376 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:43:24.826550 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:43:24.827068 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:43:24.827075 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:43:24.827420 (+ 345us) client_negotiation.cc:769] Sending connection context
0925 04:43:24.828307 (+ 887us) client_negotiation.cc:240] Negotiation successful
0925 04:43:24.828656 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"mutex_wait_us":1,"thread_start_us":133,"threads_started":1}
I20260925 04:43:24.829834 22883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:24.816893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42186 (local address 127.0.98.60:41675)
0925 04:43:24.817259 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:24.817272 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:24.817430 (+ 158us) server_negotiation.cc:407] Connection header received
0925 04:43:24.817978 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:24.818006 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:24.818243 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:24.818434 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:24.819236 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:24.820123 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:24.820942 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:24.821383 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:24.821476 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:24.824271 (+ 2795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:43:24.824278 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:43:24.824293 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:43:24.824316 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:43:24.826250 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:24.826748 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:24.826754 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:24.826756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:24.826853 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:24.827120 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:24.827123 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:24.827124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:24.827442 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:43:24.828936 (+ 1494us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:24.829046 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 04:43:24.829191 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":246,"thread_start_us":95,"threads_started":1}
Sep 25 04:43:24 dist-test-slave-dc95 krb5kdc[22647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 04:43:24.831219 22879 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32855 } has no permanent_uuid. Determining permanent_uuid...
I20260925 04:43:24.834932 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:24.824530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41675 (local address 127.0.0.1:42192)
0925 04:43:24.825343 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0925 04:43:24.825359 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:43:24.825444 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:43:24.825944 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:43:24.825948 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:43:24.826126 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:43:24.826254 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:43:24.826262 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:43:24.827278 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:43:24.827281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:43:24.828402 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:43:24.828413 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:43:24.829141 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:43:24.829145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:43:24.829234 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:24.829664 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:43:24.829678 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:43:24.831500 (+ 1822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:43:24.833383 (+ 1883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:43:24.833386 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:43:24.833388 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:43:24.833634 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:43:24.833944 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:43:24.833947 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:43:24.833949 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:43:24.834052 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:43:24.834456 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:43:24.834459 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:43:24.834545 (+ 86us) client_negotiation.cc:769] Sending connection context
0925 04:43:24.834617 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 04:43:24.834750 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135}
I20260925 04:43:24.835417 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:24.823967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41675 (local address 127.0.0.1:42190)
0925 04:43:24.824108 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 04:43:24.824123 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:43:24.824200 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:43:24.824979 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:43:24.824983 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:43:24.825146 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:43:24.825253 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:43:24.825260 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:43:24.826389 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:43:24.826392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:43:24.827133 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:43:24.827142 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:43:24.827735 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:43:24.827739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:43:24.827818 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:24.828419 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:43:24.828433 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:43:24.829948 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:43:24.833961 (+ 4013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:43:24.833964 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:43:24.833966 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:43:24.834213 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:43:24.834500 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:43:24.834503 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:43:24.834505 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:43:24.834587 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:43:24.834976 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:43:24.834979 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:43:24.835068 (+ 89us) client_negotiation.cc:769] Sending connection context
0925 04:43:24.835140 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 04:43:24.835265 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260925 04:43:24.835450 22885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:24.824573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42192 (local address 127.0.98.60:41675)
0925 04:43:24.825666 (+ 1093us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:24.825674 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:24.825686 (+ 12us) server_negotiation.cc:407] Connection header received
0925 04:43:24.825713 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:24.825716 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:24.825795 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:24.826352 (+ 557us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:24.826457 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:24.827155 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:24.828706 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:24.829004 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:24.831626 (+ 2622us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:24.831730 (+ 104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:43:24.831733 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:43:24.831735 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:43:24.831752 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:43:24.833278 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:24.833745 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:24.833748 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:24.833750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:24.833846 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:24.834162 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:24.834165 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:24.834167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:24.834356 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:43:24.835101 (+ 745us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:24.835172 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 04:43:24.835275 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1030,"thread_start_us":103,"threads_started":1}
I20260925 04:43:24.836301 22884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:24.824039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42190 (local address 127.0.98.60:41675)
0925 04:43:24.824740 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:24.824748 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:24.824761 (+ 13us) server_negotiation.cc:407] Connection header received
0925 04:43:24.824793 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:24.824795 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:24.824838 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:24.825373 (+ 535us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:24.825468 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:24.826268 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:24.827253 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:24.827586 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:24.828014 (+ 428us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:24.830090 (+ 2076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:43:24.830094 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:43:24.830097 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:43:24.830119 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:43:24.833841 (+ 3722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:24.834322 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:24.834325 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:24.834327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:24.834406 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:24.834692 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:24.834695 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:24.834697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:24.834878 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:43:24.835978 (+ 1100us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:24.836045 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 04:43:24.836145 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":103,"threads_started":1}
Sep 25 04:43:24 dist-test-slave-dc95 krb5kdc[22647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 25 04:43:24 dist-test-slave-dc95 krb5kdc[22647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:43:24.839226 22728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91f939ff44804b7b9633ff3df1c9aafe: Bootstrap starting.
I20260925 04:43:24.839226 22803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa063903041144afa6ecc2495bed0c60: Bootstrap starting.
I20260925 04:43:24.841483 22809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:24.832079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39022 (local address 127.0.98.61:32855)
0925 04:43:24.832252 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:24.832256 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:24.832270 (+ 14us) server_negotiation.cc:407] Connection header received
0925 04:43:24.832306 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:24.832309 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:24.832352 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:24.832425 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:24.833091 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:24.833858 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:24.834959 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:24.835305 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:24.835374 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:24.837952 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:43:24.837956 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:43:24.837958 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:43:24.837983 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:43:24.839639 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:24.840141 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:24.840144 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:24.840145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:24.840239 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:24.840535 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:24.840539 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:24.840540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:24.840791 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:43:24.840860 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:24.841196 (+ 336us) server_negotiation.cc:299] Negotiation successful
0925 04:43:24.841314 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260925 04:43:24.842103 22803 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa063903041144afa6ecc2495bed0c60: Neither blocks nor log segments found. Creating new log.
I20260925 04:43:24.842106 22728 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91f939ff44804b7b9633ff3df1c9aafe: Neither blocks nor log segments found. Creating new log.
I20260925 04:43:24.842262 22881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:24.831712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32855 (local address 127.0.0.1:39022)
0925 04:43:24.831809 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0925 04:43:24.831821 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:43:24.831886 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:43:24.832530 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:43:24.832533 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:43:24.832677 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:43:24.832770 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:43:24.832777 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:43:24.834076 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:43:24.834079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:43:24.834839 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:43:24.834847 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:43:24.835664 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:43:24.835668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:43:24.835768 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:24.836464 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:43:24.836478 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:43:24.837810 (+ 1332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:43:24.839812 (+ 2002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:43:24.839815 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:43:24.839817 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:43:24.840036 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:43:24.840346 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:43:24.840349 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:43:24.840350 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:43:24.840432 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:43:24.840929 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:43:24.840931 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:43:24.841023 (+ 92us) client_negotiation.cc:769] Sending connection context
0925 04:43:24.841882 (+ 859us) client_negotiation.cc:240] Negotiation successful
0925 04:43:24.842023 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 04:43:24.842439 22732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:24.832826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39858 (local address 127.0.98.62:38567)
0925 04:43:24.832928 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:24.832932 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:24.832981 (+ 49us) server_negotiation.cc:407] Connection header received
0925 04:43:24.833086 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:24.833088 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:24.833126 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:24.833190 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:24.833881 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:24.834602 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:24.835630 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:24.835930 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:24.836002 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:24.839073 (+ 3071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:43:24.839077 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:43:24.839079 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:43:24.839104 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:43:24.840628 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:24.841110 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:24.841114 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:24.841115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:24.841196 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:24.841496 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:24.841499 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:24.841501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:24.841692 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:43:24.841825 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:24.842176 (+ 351us) server_negotiation.cc:299] Negotiation successful
0925 04:43:24.842285 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 04:43:24.842979 22728 log.cc:824] T 00000000000000000000000000000000 P 91f939ff44804b7b9633ff3df1c9aafe: Log is configured to *not* fsync() on all Append() calls
I20260925 04:43:24.843135 22803 log.cc:824] T 00000000000000000000000000000000 P aa063903041144afa6ecc2495bed0c60: Log is configured to *not* fsync() on all Append() calls
I20260925 04:43:24.843364 22879 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41675 } has no permanent_uuid. Determining permanent_uuid...
I20260925 04:43:24.845434 22803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa063903041144afa6ecc2495bed0c60: No bootstrap required, opened a new log
I20260925 04:43:24.845978 22728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91f939ff44804b7b9633ff3df1c9aafe: No bootstrap required, opened a new log
Sep 25 04:43:24 dist-test-slave-dc95 krb5kdc[22647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 25 04:43:24 dist-test-slave-dc95 krb5kdc[22647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 04:43:24.849632 22803 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa063903041144afa6ecc2495bed0c60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f939ff44804b7b9633ff3df1c9aafe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38567 } } peers { permanent_uuid: "aa063903041144afa6ecc2495bed0c60" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32855 } } peers { permanent_uuid: "3090682f59564033a4cd064e53f2d85d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41675 } }
I20260925 04:43:24.849895 22803 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa063903041144afa6ecc2495bed0c60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa063903041144afa6ecc2495bed0c60, State: Initialized, Role: FOLLOWER
I20260925 04:43:24.850085 22728 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91f939ff44804b7b9633ff3df1c9aafe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f939ff44804b7b9633ff3df1c9aafe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38567 } } peers { permanent_uuid: "aa063903041144afa6ecc2495bed0c60" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32855 } } peers { permanent_uuid: "3090682f59564033a4cd064e53f2d85d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41675 } }
I20260925 04:43:24.850261 22803 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa063903041144afa6ecc2495bed0c60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f939ff44804b7b9633ff3df1c9aafe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38567 } } peers { permanent_uuid: "aa063903041144afa6ecc2495bed0c60" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32855 } } peers { permanent_uuid: "3090682f59564033a4cd064e53f2d85d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41675 } }
I20260925 04:43:24.850531 22728 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91f939ff44804b7b9633ff3df1c9aafe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91f939ff44804b7b9633ff3df1c9aafe, State: Initialized, Role: FOLLOWER
I20260925 04:43:24.850862 22728 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91f939ff44804b7b9633ff3df1c9aafe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f939ff44804b7b9633ff3df1c9aafe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38567 } } peers { permanent_uuid: "aa063903041144afa6ecc2495bed0c60" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32855 } } peers { permanent_uuid: "3090682f59564033a4cd064e53f2d85d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41675 } }
I20260925 04:43:24.851853 22803 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa063903041144afa6ecc2495bed0c60 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:43:24.852190 22728 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91f939ff44804b7b9633ff3df1c9aafe [sys.catalog]: configured and running, proceeding with master startup.
W20260925 04:43:24.853240 22681 tablet.cc:2369] T 00000000000000000000000000000000 P 91f939ff44804b7b9633ff3df1c9aafe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:43:24.853264 22881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:24.843863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41675 (local address 127.0.0.1:42196)
0925 04:43:24.844067 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 04:43:24.844079 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:43:24.844150 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:43:24.844590 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:43:24.844593 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:43:24.844764 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:43:24.844856 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:43:24.844863 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:43:24.845823 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:43:24.845826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:43:24.846570 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:43:24.846579 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:43:24.847211 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:43:24.847214 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:43:24.847289 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:24.847860 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:43:24.847874 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:43:24.849396 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:43:24.851811 (+ 2415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:43:24.851814 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:43:24.851816 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:43:24.852033 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:43:24.852284 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:43:24.852286 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:43:24.852288 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:43:24.852380 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:43:24.852794 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:43:24.852796 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:43:24.852873 (+ 77us) client_negotiation.cc:769] Sending connection context
0925 04:43:24.852960 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 04:43:24.853101 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138}
I20260925 04:43:24.853793 22884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:24.844059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42196 (local address 127.0.98.60:41675)
0925 04:43:24.844179 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:24.844183 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:24.844213 (+ 30us) server_negotiation.cc:407] Connection header received
0925 04:43:24.844246 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:24.844249 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:24.844500 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:24.844576 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:24.844967 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:24.845723 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:24.846680 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:24.847015 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:24.847100 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:24.849533 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:43:24.849537 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:43:24.849540 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:43:24.849561 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:43:24.851699 (+ 2138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:24.852126 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:24.852129 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:24.852131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:24.852202 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:24.852468 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:24.852470 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:24.852472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:24.852685 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:43:24.852764 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:24.853532 (+ 768us) server_negotiation.cc:299] Negotiation successful
0925 04:43:24.853647 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 04:43:24.854411 22809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:24.844339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39028 (local address 127.0.98.61:32855)
0925 04:43:24.844439 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:24.844443 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:24.844454 (+ 11us) server_negotiation.cc:407] Connection header received
0925 04:43:24.844562 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:24.844565 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:24.844607 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:24.844672 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:24.845448 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:24.846148 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:24.847156 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:24.847453 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:24.847525 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:24.850248 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:43:24.850252 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:43:24.850255 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:43:24.850282 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:43:24.851820 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:24.852310 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:24.852313 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:24.852315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:24.852387 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:24.852655 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:24.852658 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:24.852660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:24.852869 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:43:24.853064 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:24.854164 (+ 1100us) server_negotiation.cc:299] Negotiation successful
0925 04:43:24.854279 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 04:43:24.854756 22889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91f939ff44804b7b9633ff3df1c9aafe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f939ff44804b7b9633ff3df1c9aafe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38567 } } peers { permanent_uuid: "aa063903041144afa6ecc2495bed0c60" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32855 } } peers { permanent_uuid: "3090682f59564033a4cd064e53f2d85d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41675 } } }
I20260925 04:43:24.854938 22889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91f939ff44804b7b9633ff3df1c9aafe [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:43:24.856880 22879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3090682f59564033a4cd064e53f2d85d: Bootstrap starting.
I20260925 04:43:24.858698 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260925 04:43:24.859596 22911 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aa063903041144afa6ecc2495bed0c60: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 04:43:24.859705 22911 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 04:43:24.859714 22879 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3090682f59564033a4cd064e53f2d85d: Neither blocks nor log segments found. Creating new log.
I20260925 04:43:24.860580 22879 log.cc:824] T 00000000000000000000000000000000 P 3090682f59564033a4cd064e53f2d85d: Log is configured to *not* fsync() on all Append() calls
I20260925 04:43:24.860617 22702 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "91f939ff44804b7b9633ff3df1c9aafe"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39858
I20260925 04:43:24.860833 22702 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91f939ff44804b7b9633ff3df1c9aafe [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 04:43:24.860927 22702 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91f939ff44804b7b9633ff3df1c9aafe [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:43:24.862253 22702 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91f939ff44804b7b9633ff3df1c9aafe [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f939ff44804b7b9633ff3df1c9aafe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38567 } } peers { permanent_uuid: "aa063903041144afa6ecc2495bed0c60" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32855 } } peers { permanent_uuid: "3090682f59564033a4cd064e53f2d85d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41675 } }
I20260925 04:43:24.862277 22888 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa063903041144afa6ecc2495bed0c60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f939ff44804b7b9633ff3df1c9aafe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38567 } } peers { permanent_uuid: "aa063903041144afa6ecc2495bed0c60" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32855 } } peers { permanent_uuid: "3090682f59564033a4cd064e53f2d85d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41675 } } }
I20260925 04:43:24.862382 22888 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa063903041144afa6ecc2495bed0c60 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:43:24.862905 22879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3090682f59564033a4cd064e53f2d85d: No bootstrap required, opened a new log
I20260925 04:43:24.863379 22702 leader_election.cc:288] T 00000000000000000000000000000000 P 91f939ff44804b7b9633ff3df1c9aafe [CANDIDATE]: Term 1 election: Requested vote from peers aa063903041144afa6ecc2495bed0c60 (127.0.98.61:32855), 3090682f59564033a4cd064e53f2d85d (127.0.98.60:41675)
I20260925 04:43:24.860636 22660 master_runner.cc:186] Error getting master registration for 127.0.98.60:41675: OK, instance_id { permanent_uuid: "3090682f59564033a4cd064e53f2d85d" instance_seqno: 1790311404803768 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 04:43:24.864463 22660 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 04:43:24.864439 22777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "91f939ff44804b7b9633ff3df1c9aafe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "aa063903041144afa6ecc2495bed0c60"
I20260925 04:43:24.864666 22777 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa063903041144afa6ecc2495bed0c60 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:43:24.865993 22777 raft_consensus.cc:2454] T 00000000000000000000000000000000 P aa063903041144afa6ecc2495bed0c60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91f939ff44804b7b9633ff3df1c9aafe in term 1.
I20260925 04:43:24.866463 22676 leader_election.cc:302] T 00000000000000000000000000000000 P 91f939ff44804b7b9633ff3df1c9aafe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91f939ff44804b7b9633ff3df1c9aafe, aa063903041144afa6ecc2495bed0c60; no voters:
I20260925 04:43:24.866865 22889 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91f939ff44804b7b9633ff3df1c9aafe [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:43:24.867154 22889 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91f939ff44804b7b9633ff3df1c9aafe [term 1 LEADER]: Becoming Leader. State: Replica: 91f939ff44804b7b9633ff3df1c9aafe, State: Running, Role: LEADER
I20260925 04:43:24.867204 22879 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3090682f59564033a4cd064e53f2d85d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f939ff44804b7b9633ff3df1c9aafe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38567 } } peers { permanent_uuid: "aa063903041144afa6ecc2495bed0c60" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32855 } } peers { permanent_uuid: "3090682f59564033a4cd064e53f2d85d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41675 } }
I20260925 04:43:24.867625 22879 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3090682f59564033a4cd064e53f2d85d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3090682f59564033a4cd064e53f2d85d, State: Initialized, Role: FOLLOWER
I20260925 04:43:24.867667 22889 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91f939ff44804b7b9633ff3df1c9aafe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f939ff44804b7b9633ff3df1c9aafe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38567 } } peers { permanent_uuid: "aa063903041144afa6ecc2495bed0c60" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32855 } } peers { permanent_uuid: "3090682f59564033a4cd064e53f2d85d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41675 } }
I20260925 04:43:24.867969 22879 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3090682f59564033a4cd064e53f2d85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f939ff44804b7b9633ff3df1c9aafe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38567 } } peers { permanent_uuid: "aa063903041144afa6ecc2495bed0c60" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32855 } } peers { permanent_uuid: "3090682f59564033a4cd064e53f2d85d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41675 } }
I20260925 04:43:24.869369 22879 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3090682f59564033a4cd064e53f2d85d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:43:24.869513 22853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "91f939ff44804b7b9633ff3df1c9aafe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3090682f59564033a4cd064e53f2d85d"
I20260925 04:43:24.869798 22853 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3090682f59564033a4cd064e53f2d85d [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:43:24.869746 22889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91f939ff44804b7b9633ff3df1c9aafe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91f939ff44804b7b9633ff3df1c9aafe. Latest consensus state: current_term: 1 leader_uuid: "91f939ff44804b7b9633ff3df1c9aafe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f939ff44804b7b9633ff3df1c9aafe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38567 } } peers { permanent_uuid: "aa063903041144afa6ecc2495bed0c60" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32855 } } peers { permanent_uuid: "3090682f59564033a4cd064e53f2d85d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41675 } } }
I20260925 04:43:24.869944 22889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91f939ff44804b7b9633ff3df1c9aafe [sys.catalog]: This master's current role is: LEADER
I20260925 04:43:24.870370 22922 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:43:24.871196 22853 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3090682f59564033a4cd064e53f2d85d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91f939ff44804b7b9633ff3df1c9aafe in term 1.
I20260925 04:43:24.867933 22735 master_runner.cc:186] Error getting master registration for 127.0.98.60:41675: OK, instance_id { permanent_uuid: "3090682f59564033a4cd064e53f2d85d" instance_seqno: 1790311404803768 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 04:43:24.872615 22735 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260925 04:43:24.874953 22932 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3090682f59564033a4cd064e53f2d85d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 04:43:24.875052 22932 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 04:43:24.875428 22917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3090682f59564033a4cd064e53f2d85d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f939ff44804b7b9633ff3df1c9aafe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38567 } } peers { permanent_uuid: "aa063903041144afa6ecc2495bed0c60" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32855 } } peers { permanent_uuid: "3090682f59564033a4cd064e53f2d85d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41675 } } }
I20260925 04:43:24.875571 22917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3090682f59564033a4cd064e53f2d85d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:43:24.877326 22922 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:43:24.886487 22732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:24.872840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39860 (local address 127.0.98.62:38567)
0925 04:43:24.873197 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:24.873202 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:24.876824 (+ 3622us) server_negotiation.cc:407] Connection header received
0925 04:43:24.876965 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:24.876968 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:24.877019 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:24.877102 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:24.879878 (+ 2776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:24.880538 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:24.881494 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:24.881791 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:24.881856 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:24.883459 (+ 1603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:43:24.883463 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:43:24.883466 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:43:24.883496 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:43:24.884869 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:24.885313 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:24.885316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:24.885318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:24.885402 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:24.885650 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:24.885660 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:24.885662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:24.885860 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:43:24.885974 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:24.886210 (+ 236us) server_negotiation.cc:299] Negotiation successful
0925 04:43:24.886325 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280}
I20260925 04:43:24.894362 22809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:24.876002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39038 (local address 127.0.98.61:32855)
0925 04:43:24.876121 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:24.876126 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:24.886696 (+ 10570us) server_negotiation.cc:407] Connection header received
0925 04:43:24.886883 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:24.886886 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:24.886940 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:24.886992 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:24.887845 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:24.888385 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:24.889129 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:24.889348 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:24.889439 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:24.890657 (+ 1218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:43:24.890661 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:43:24.890664 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:43:24.890692 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:43:24.892121 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:24.892660 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:24.892664 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:24.892666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:24.892757 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:24.893043 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:24.893046 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:24.893048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:24.893279 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:43:24.893426 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:24.894065 (+ 639us) server_negotiation.cc:299] Negotiation successful
0925 04:43:24.894194 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 04:43:24.903910 22853 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3090682f59564033a4cd064e53f2d85d [term 1 FOLLOWER]: Refusing update from remote peer 91f939ff44804b7b9633ff3df1c9aafe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:43:24.903898 22777 raft_consensus.cc:1261] T 00000000000000000000000000000000 P aa063903041144afa6ecc2495bed0c60 [term 1 FOLLOWER]: Refusing update from remote peer 91f939ff44804b7b9633ff3df1c9aafe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:43:24.904670 22889 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 91f939ff44804b7b9633ff3df1c9aafe [LEADER]: Connected to new peer: Peer: permanent_uuid: "3090682f59564033a4cd064e53f2d85d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:43:24.904994 22914 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 91f939ff44804b7b9633ff3df1c9aafe [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa063903041144afa6ecc2495bed0c60" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:43:24.909162 22884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:24.876467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42202 (local address 127.0.98.60:41675)
0925 04:43:24.876571 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:24.876575 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:24.891916 (+ 15341us) server_negotiation.cc:407] Connection header received
0925 04:43:24.892109 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:24.892112 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:24.892162 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:24.892211 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:24.893073 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:24.893887 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:24.895150 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:24.895498 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:24.895570 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:24.897733 (+ 2163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:43:24.897737 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:43:24.897740 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:43:24.897771 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:43:24.899437 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:24.899963 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:24.899967 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:24.899969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:24.900056 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:24.907720 (+ 7664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:24.907725 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:24.907727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:24.908034 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:43:24.908424 (+ 390us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:24.908849 (+ 425us) server_negotiation.cc:299] Negotiation successful
0925 04:43:24.908994 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 04:43:24.913129 22888 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa063903041144afa6ecc2495bed0c60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91f939ff44804b7b9633ff3df1c9aafe. Latest consensus state: current_term: 1 leader_uuid: "91f939ff44804b7b9633ff3df1c9aafe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f939ff44804b7b9633ff3df1c9aafe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38567 } } peers { permanent_uuid: "aa063903041144afa6ecc2495bed0c60" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32855 } } peers { permanent_uuid: "3090682f59564033a4cd064e53f2d85d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41675 } } }
I20260925 04:43:24.913307 22888 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa063903041144afa6ecc2495bed0c60 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:43:24.914232 22917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3090682f59564033a4cd064e53f2d85d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91f939ff44804b7b9633ff3df1c9aafe. Latest consensus state: current_term: 1 leader_uuid: "91f939ff44804b7b9633ff3df1c9aafe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f939ff44804b7b9633ff3df1c9aafe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38567 } } peers { permanent_uuid: "aa063903041144afa6ecc2495bed0c60" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32855 } } peers { permanent_uuid: "3090682f59564033a4cd064e53f2d85d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41675 } } }
I20260925 04:43:24.914376 22917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3090682f59564033a4cd064e53f2d85d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:43:24.924407 22938 mvcc.cc:205] Tried to move back new op lower bound from 7333115514476937216 to 7333115514342334464. Current Snapshot: MvccSnapshot[applied={T|T < 7333115514476937216}]
I20260925 04:43:24.925148 22889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91f939ff44804b7b9633ff3df1c9aafe [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "91f939ff44804b7b9633ff3df1c9aafe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f939ff44804b7b9633ff3df1c9aafe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38567 } } peers { permanent_uuid: "aa063903041144afa6ecc2495bed0c60" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32855 } } peers { permanent_uuid: "3090682f59564033a4cd064e53f2d85d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41675 } } }
I20260925 04:43:24.925283 22889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91f939ff44804b7b9633ff3df1c9aafe [sys.catalog]: This master's current role is: LEADER
I20260925 04:43:24.926568 22888 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa063903041144afa6ecc2495bed0c60 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "91f939ff44804b7b9633ff3df1c9aafe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f939ff44804b7b9633ff3df1c9aafe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38567 } } peers { permanent_uuid: "aa063903041144afa6ecc2495bed0c60" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32855 } } peers { permanent_uuid: "3090682f59564033a4cd064e53f2d85d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41675 } } }
I20260925 04:43:24.926709 22888 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa063903041144afa6ecc2495bed0c60 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:43:24.927201 22941 mvcc.cc:205] Tried to move back new op lower bound from 7333115514476937216 to 7333115514342334464. Current Snapshot: MvccSnapshot[applied={T|T < 7333115514476937216}]
I20260925 04:43:24.930998 22922 catalog_manager.cc:1347] Generated new cluster ID: c503d18242f34fe1962492083401fcdf
I20260925 04:43:24.931080 22922 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:43:24.937655 22914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91f939ff44804b7b9633ff3df1c9aafe [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "91f939ff44804b7b9633ff3df1c9aafe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f939ff44804b7b9633ff3df1c9aafe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38567 } } peers { permanent_uuid: "aa063903041144afa6ecc2495bed0c60" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32855 } } peers { permanent_uuid: "3090682f59564033a4cd064e53f2d85d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41675 } } }
I20260925 04:43:24.937794 22914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91f939ff44804b7b9633ff3df1c9aafe [sys.catalog]: This master's current role is: LEADER
I20260925 04:43:24.939052 22940 mvcc.cc:205] Tried to move back new op lower bound from 7333115514476937216 to 7333115514342334464. Current Snapshot: MvccSnapshot[applied={T|T < 7333115514476937216}]
I20260925 04:43:24.939211 22917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3090682f59564033a4cd064e53f2d85d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "91f939ff44804b7b9633ff3df1c9aafe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f939ff44804b7b9633ff3df1c9aafe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38567 } } peers { permanent_uuid: "aa063903041144afa6ecc2495bed0c60" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32855 } } peers { permanent_uuid: "3090682f59564033a4cd064e53f2d85d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41675 } } }
I20260925 04:43:24.939311 22917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3090682f59564033a4cd064e53f2d85d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:43:24.980775 22922 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:43:24.982149 22922 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:43:25.005241 22922 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 91f939ff44804b7b9633ff3df1c9aafe: Generated new TSK 0
I20260925 04:43:25.006208 22922 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 04:43:25.042435 22732 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:43:25.038790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39874 (local address 127.0.98.62:38567)
0925 04:43:25.038930 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:25.038935 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:25.041192 (+ 2257us) server_negotiation.cc:407] Connection header received
0925 04:43:25.041421 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:25.041426 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:25.041483 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:25.041541 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:25.042278 (+ 737us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39874: BlockingRecv error: recv got EOF from 127.0.0.1:39874 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260925 04:43:25.044793 22809 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:43:25.040496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39050 (local address 127.0.98.61:32855)
0925 04:43:25.040633 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:25.040637 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:25.043273 (+ 2636us) server_negotiation.cc:407] Connection header received
0925 04:43:25.043570 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:25.043574 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:25.043685 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:25.043748 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:25.044651 (+ 903us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39050: BlockingRecv error: recv got EOF from 127.0.0.1:39050 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260925 04:43:25.045670 22884 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:43:25.039726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42208 (local address 127.0.98.60:41675)
0925 04:43:25.039855 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:25.039860 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:25.043054 (+ 3194us) server_negotiation.cc:407] Connection header received
0925 04:43:25.043931 (+ 877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:25.043935 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:25.043992 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:25.044045 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:25.045520 (+ 1475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42208: BlockingRecv error: recv got EOF from 127.0.0.1:42208 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260925 04:43:25.051244 22809 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:43:25.049401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39056 (local address 127.0.98.61:32855)
0925 04:43:25.049524 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:25.049529 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:25.050038 (+ 509us) server_negotiation.cc:407] Connection header received
0925 04:43:25.050320 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:25.050324 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:25.050405 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:25.050460 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:25.051097 (+ 637us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39056: BlockingRecv error: recv got EOF from 127.0.0.1:39056 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260925 04:43:25.057611 22809 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:43:25.055697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39072 (local address 127.0.98.61:32855)
0925 04:43:25.055843 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:25.055848 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:25.056441 (+ 593us) server_negotiation.cc:407] Connection header received
0925 04:43:25.056696 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:25.056700 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:25.056763 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:25.056821 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:25.057452 (+ 631us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39072: BlockingRecv error: recv got EOF from 127.0.0.1:39072 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260925 04:43:25.063830 22809 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:43:25.062158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39076 (local address 127.0.98.61:32855)
0925 04:43:25.062275 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:25.062279 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:25.062784 (+ 505us) server_negotiation.cc:407] Connection header received
0925 04:43:25.062968 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:25.062972 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:25.063024 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:25.063072 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:25.063705 (+ 633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39076: BlockingRecv error: recv got EOF from 127.0.0.1:39076 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260925 04:43:25.072311 22809 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:43:25.070461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39084 (local address 127.0.98.61:32855)
0925 04:43:25.070673 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:25.070677 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:25.071195 (+ 518us) server_negotiation.cc:407] Connection header received
0925 04:43:25.071398 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:25.071401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:25.071454 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:25.071505 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:25.072179 (+ 674us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39084: BlockingRecv error: recv got EOF from 127.0.0.1:39084 (error 108)
Metrics: {"server-negotiator.queue_time_us":133}
W20260925 04:43:25.084829 22809 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:43:25.083017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39098 (local address 127.0.98.61:32855)
0925 04:43:25.083151 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:25.083155 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:25.083794 (+ 639us) server_negotiation.cc:407] Connection header received
0925 04:43:25.084017 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:25.084021 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:25.084079 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:25.084130 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:25.084707 (+ 577us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39098: BlockingRecv error: recv got EOF from 127.0.0.1:39098 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260925 04:43:25.104816 22809 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:43:25.103341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39114 (local address 127.0.98.61:32855)
0925 04:43:25.103477 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:25.103482 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:25.103739 (+ 257us) server_negotiation.cc:407] Connection header received
0925 04:43:25.103964 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:25.103968 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:25.104043 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:25.104105 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:25.104692 (+ 587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39114: BlockingRecv error: recv got EOF from 127.0.0.1:39114 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260925 04:43:25.140415 22809 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:43:25.139164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39124 (local address 127.0.98.61:32855)
0925 04:43:25.139306 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:25.139310 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:25.139504 (+ 194us) server_negotiation.cc:407] Connection header received
0925 04:43:25.139704 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:25.139708 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:25.139762 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:25.139839 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:25.140301 (+ 462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39124: BlockingRecv error: recv got EOF from 127.0.0.1:39124 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260925 04:43:25.207713 22809 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:43:25.206489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39138 (local address 127.0.98.61:32855)
0925 04:43:25.206616 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:25.206630 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:25.206755 (+ 125us) server_negotiation.cc:407] Connection header received
0925 04:43:25.206879 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:25.206881 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:25.206932 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:25.207017 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:25.207526 (+ 509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39138: BlockingRecv error: recv got EOF from 127.0.0.1:39138 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260925 04:43:25.338776 22809 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:43:25.337678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39148 (local address 127.0.98.61:32855)
0925 04:43:25.337818 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:25.337821 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:25.337928 (+ 107us) server_negotiation.cc:407] Connection header received
0925 04:43:25.338070 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:25.338073 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:25.338121 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:25.338183 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:25.338650 (+ 467us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39148: BlockingRecv error: recv got EOF from 127.0.0.1:39148 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260925 04:43:25.597791 22809 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:43:25.596921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39158 (local address 127.0.98.61:32855)
0925 04:43:25.597043 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:25.597046 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:25.597083 (+ 37us) server_negotiation.cc:407] Connection header received
0925 04:43:25.597136 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:25.597139 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:25.597184 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:25.597256 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:25.597670 (+ 414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39158: BlockingRecv error: recv got EOF from 127.0.0.1:39158 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 04:43:25.861204 22911 catalog_manager.cc:1259] Loaded cluster ID: c503d18242f34fe1962492083401fcdf
I20260925 04:43:25.861301 22911 catalog_manager.cc:1552] T 00000000000000000000000000000000 P aa063903041144afa6ecc2495bed0c60: loading cluster ID for follower catalog manager: success
I20260925 04:43:25.863474 22911 catalog_manager.cc:1574] T 00000000000000000000000000000000 P aa063903041144afa6ecc2495bed0c60: acquiring CA information for follower catalog manager: success
I20260925 04:43:25.864850 22911 catalog_manager.cc:1602] T 00000000000000000000000000000000 P aa063903041144afa6ecc2495bed0c60: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 04:43:25.876410 22932 catalog_manager.cc:1259] Loaded cluster ID: c503d18242f34fe1962492083401fcdf
I20260925 04:43:25.876495 22932 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3090682f59564033a4cd064e53f2d85d: loading cluster ID for follower catalog manager: success
I20260925 04:43:25.878173 22932 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3090682f59564033a4cd064e53f2d85d: acquiring CA information for follower catalog manager: success
I20260925 04:43:25.879383 22932 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3090682f59564033a4cd064e53f2d85d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 04:43:26.116358 23011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:26.111963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39164 (local address 127.0.98.61:32855)
0925 04:43:26.112312 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:26.112316 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:26.112335 (+ 19us) server_negotiation.cc:407] Connection header received
0925 04:43:26.112388 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:26.112391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:26.112512 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:26.112620 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:43:26.113006 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:26.113606 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:26.114600 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:26.114850 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:26.114951 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:26.115313 (+ 362us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:43:26.115870 (+ 557us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:43:26.115972 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:26.116077 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 04:43:26.116140 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":76,"threads_started":1}
I20260925 04:43:26.123843 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:26.119659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42214 (local address 127.0.98.60:41675)
0925 04:43:26.119929 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:26.119933 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:26.119947 (+ 14us) server_negotiation.cc:407] Connection header received
0925 04:43:26.120027 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:26.120029 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:26.120228 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:26.120357 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:43:26.120596 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:26.121290 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:26.122223 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:26.122453 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:26.122542 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:26.122871 (+ 329us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:43:26.123353 (+ 482us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:43:26.123461 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:26.123577 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 04:43:26.123677 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":100,"threads_started":1}
I20260925 04:43:26.131371 23011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:26.127025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39176 (local address 127.0.98.61:32855)
0925 04:43:26.127208 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:26.127215 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:26.127434 (+ 219us) server_negotiation.cc:407] Connection header received
0925 04:43:26.127704 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:26.127708 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:26.127768 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:26.127838 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:43:26.128189 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:26.128896 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:26.130225 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:26.130541 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:26.130640 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:26.130873 (+ 233us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:43:26.130955 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:43:26.131015 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:26.131146 (+ 131us) server_negotiation.cc:299] Negotiation successful
0925 04:43:26.131216 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 04:43:26.131487 23024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:26.127045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39884 (local address 127.0.98.62:38567)
0925 04:43:26.127711 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:26.127715 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:26.127728 (+ 13us) server_negotiation.cc:407] Connection header received
0925 04:43:26.127856 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:26.127859 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:26.128028 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:26.128173 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:43:26.128525 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:26.129213 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:26.130034 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:26.130311 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:26.130403 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:26.130691 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:43:26.131051 (+ 360us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:43:26.131128 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:26.131280 (+ 152us) server_negotiation.cc:299] Negotiation successful
0925 04:43:26.131340 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":63,"threads_started":1}
I20260925 04:43:26.132040 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:26.127472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42226 (local address 127.0.98.60:41675)
0925 04:43:26.127599 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:26.127632 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:26.127705 (+ 73us) server_negotiation.cc:407] Connection header received
0925 04:43:26.127889 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:26.127892 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:26.127942 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:26.128002 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:43:26.128256 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:26.128848 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:26.130722 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:26.131015 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:26.131117 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:26.131382 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:43:26.131466 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:43:26.131514 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:26.131851 (+ 337us) server_negotiation.cc:299] Negotiation successful
0925 04:43:26.131918 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 04:43:26.141781 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 22660
I20260925 04:43:26.147564 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 22735
I20260925 04:43:26.153113 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 22810
2026-09-25T04:43:26Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3069 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:43:26 dist-test-slave-dc95 krb5kdc[23028](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:43:26 dist-test-slave-dc95 krb5kdc[23028](info): set up 2 sockets
Sep 25 04:43:26 dist-test-slave-dc95 krb5kdc[23028](info): commencing operation
krb5kdc: starting...
W20260925 04:43:27.111871 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/krb5kdc/test-user.keytab.
Sep 25 04:43:27 dist-test-slave-dc95 krb5kdc[23028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:43:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:43:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 04:43:27.235808 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45775
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45775,127.0.98.61:46375,127.0.98.60:42351
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:43:27.348096 23041 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 04:43:27.348416 23041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:43:27.348461 23041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:43:27.348526 23041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:43:27.352641 23041 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:43:27.352712 23041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:43:27.352738 23041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:43:27.352761 23041 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:43:27.352792 23041 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:43:27.358336 23041 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45775,127.0.98.61:46375,127.0.98.60:42351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23041
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:43:27.359452 23041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:43:27.360509 23041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:43:27.369026 23047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:43:27.369026 23049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:43:27.369026 23046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:43:27.369309 23041 server_base.cc:1034] running on GCE node
I20260925 04:43:27.369974 23041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:43:27.371125 23041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:43:27.372301 23041 hybrid_clock.cc:648] HybridClock initialized: now 1790311407372286 us; error 31 us; skew 500 ppm
Sep 25 04:43:27 dist-test-slave-dc95 krb5kdc[23028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311407, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:43:27.375099 23041 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 04:43:27.376046 23041 webserver.cc:460] Webserver started at http://127.0.98.62:46345/ using document root <none> and password file <none>
I20260925 04:43:27.376616 23041 fs_manager.cc:359] Metadata directory not provided
I20260925 04:43:27.376725 23041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:43:27.376930 23041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:43:27.379025 23041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/minicluster-data/master-0/data/instance:
uuid: "05ae38ec601d45f690d0d89d0bb754c9"
format_stamp: "Formatted at 2026-09-25 04:43:27 on dist-test-slave-dc95"
server_key: "446094904d3cfc89b2fc45374dd226a6"
server_key_iv: "7520af7a34f7b930a77ee7cb2249e312"
server_key_version: "encryptionkey@0"
I20260925 04:43:27.379594 23041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/minicluster-data/master-0/wal/instance:
uuid: "05ae38ec601d45f690d0d89d0bb754c9"
format_stamp: "Formatted at 2026-09-25 04:43:27 on dist-test-slave-dc95"
server_key: "446094904d3cfc89b2fc45374dd226a6"
server_key_iv: "7520af7a34f7b930a77ee7cb2249e312"
server_key_version: "encryptionkey@0"
I20260925 04:43:27.383396 23041 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260925 04:43:27.386109 23056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:43:27.387439 23041 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:43:27.387567 23041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/minicluster-data/master-0/wal
uuid: "05ae38ec601d45f690d0d89d0bb754c9"
format_stamp: "Formatted at 2026-09-25 04:43:27 on dist-test-slave-dc95"
server_key: "446094904d3cfc89b2fc45374dd226a6"
server_key_iv: "7520af7a34f7b930a77ee7cb2249e312"
server_key_version: "encryptionkey@0"
I20260925 04:43:27.387723 23041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:43:27.400775 23041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:43:27.404305 23041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:43:27.404501 23041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:43:27.411000 23041 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45775
I20260925 04:43:27.411269 23108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45775 every 8 connection(s)
I20260925 04:43:27.412037 23041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/minicluster-data/master-0/data/info.pb
I20260925 04:43:27.414994 23109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:43:27.419117 23109 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45775 } has no permanent_uuid. Determining permanent_uuid...
I20260925 04:43:27.422194 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 23041
I20260925 04:43:27.422331 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/minicluster-data/master-0/wal/instance
I20260925 04:43:27.422719 392 external_mini_cluster.cc:1428] Setting key 6e4abeba6716d6a398d66f1d67f80c8c
Sep 25 04:43:27 dist-test-slave-dc95 krb5kdc[23028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311407, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260925 04:43:27.436324 23112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:27.421725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38732 (local address 127.0.98.62:45775)
0925 04:43:27.422478 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:27.422492 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:27.422578 (+ 86us) server_negotiation.cc:407] Connection header received
0925 04:43:27.423763 (+ 1185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:27.423769 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:27.424050 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:27.424266 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:27.425379 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:27.426309 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:27.427816 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:27.428224 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:27.428362 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:27.431169 (+ 2807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:43:27.431175 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:43:27.431186 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:43:27.431211 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:43:27.433036 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:27.433514 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:27.433520 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:27.433522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:27.433670 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:27.434010 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:27.434013 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:27.434014 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:27.434447 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:43:27.434554 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:27.435011 (+ 457us) server_negotiation.cc:299] Negotiation successful
0925 04:43:27.435297 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"spinlock_wait_cycles":80640,"thread_start_us":181,"threads_started":1}
I20260925 04:43:27.436328 23111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:27.421725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45775 (local address 127.0.0.1:38732)
0925 04:43:27.422409 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0925 04:43:27.422466 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 04:43:27.423389 (+ 923us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:43:27.424314 (+ 925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:43:27.424320 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:43:27.424812 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:43:27.425182 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:43:27.425202 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:43:27.426444 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:43:27.426448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:43:27.427696 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:43:27.427708 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:43:27.428393 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:43:27.428397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:43:27.428537 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:27.429147 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:43:27.429168 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:43:27.430982 (+ 1814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:43:27.433144 (+ 2162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:43:27.433152 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:43:27.433161 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:43:27.433396 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:43:27.433774 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:43:27.433777 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:43:27.433779 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:43:27.433903 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:43:27.434566 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:43:27.434573 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:43:27.434755 (+ 182us) client_negotiation.cc:769] Sending connection context
0925 04:43:27.434938 (+ 183us) client_negotiation.cc:240] Negotiation successful
0925 04:43:27.435581 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"mutex_wait_us":255,"thread_start_us":181,"threads_started":1}
I20260925 04:43:27.439054 23109 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46375 } has no permanent_uuid. Determining permanent_uuid...
I20260925 04:43:27.442610 23111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:27.439671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46375 (local address 127.0.0.1:40152)
0925 04:43:27.442432 (+ 2761us) negotiation.cc:107] Waiting for socket to connect
0925 04:43:27.442522 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2656}
W20260925 04:43:27.442911 23060 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46375: connect: Connection refused (error 111)
I20260925 04:43:27.444062 23111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:27.443815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46375 (local address 127.0.0.1:40164)
0925 04:43:27.443931 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 04:43:27.443992 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 04:43:27.444233 23109 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46375: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46375: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260925 04:43:27.451188 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:46375
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45775,127.0.98.61:46375,127.0.98.60:42351
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:43:27.493474 23109 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46375 } attempt: 1
I20260925 04:43:27.494378 23111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:27.494073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46375 (local address 127.0.0.1:40180)
0925 04:43:27.494233 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 04:43:27.494298 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260925 04:43:27.495072 23111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:27.494855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46375 (local address 127.0.0.1:40192)
0925 04:43:27.494960 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0925 04:43:27.495009 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260925 04:43:27.495223 23109 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46375: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46375: connect: Connection refused (error 111)
I20260925 04:43:27.563412 23109 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46375 } attempt: 2
W20260925 04:43:27.563432 23116 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 04:43:27.563823 23116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:43:27.563868 23116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:43:27.563934 23116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 04:43:27.564318 23111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:27.564033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46375 (local address 127.0.0.1:40198)
0925 04:43:27.564174 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 04:43:27.564238 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 04:43:27.565016 23111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:27.564778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46375 (local address 127.0.0.1:40202)
0925 04:43:27.564880 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0925 04:43:27.564933 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 04:43:27.565464 23109 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46375: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46375: connect: Connection refused (error 111)
W20260925 04:43:27.568315 23116 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:43:27.568378 23116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:43:27.568397 23116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:43:27.568413 23116 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:43:27.568427 23116 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:43:27.574009 23116 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45775,127.0.98.61:46375,127.0.98.60:42351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:46375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:43:27.575163 23116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:43:27.576300 23116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:43:27.585489 23124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:43:27.585518 23121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:43:27.585574 23122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:43:27.585561 23116 server_base.cc:1034] running on GCE node
I20260925 04:43:27.586331 23116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:43:27.587535 23116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:43:27.588738 23116 hybrid_clock.cc:648] HybridClock initialized: now 1790311407588713 us; error 40 us; skew 500 ppm
Sep 25 04:43:27 dist-test-slave-dc95 krb5kdc[23028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311407, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:43:27.591897 23116 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260925 04:43:27.592940 23116 webserver.cc:460] Webserver started at http://127.0.98.61:41549/ using document root <none> and password file <none>
I20260925 04:43:27.593526 23116 fs_manager.cc:359] Metadata directory not provided
I20260925 04:43:27.593590 23116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:43:27.593792 23116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:43:27.596061 23116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/minicluster-data/master-1/data/instance:
uuid: "71d57ede7cf44d55bcf81e6677871726"
format_stamp: "Formatted at 2026-09-25 04:43:27 on dist-test-slave-dc95"
server_key: "bd09bd2b780e20f4af1d24f612216d5f"
server_key_iv: "8c724dba84b5223af88d74fe4e6d8df4"
server_key_version: "encryptionkey@0"
I20260925 04:43:27.596652 23116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/minicluster-data/master-1/wal/instance:
uuid: "71d57ede7cf44d55bcf81e6677871726"
format_stamp: "Formatted at 2026-09-25 04:43:27 on dist-test-slave-dc95"
server_key: "bd09bd2b780e20f4af1d24f612216d5f"
server_key_iv: "8c724dba84b5223af88d74fe4e6d8df4"
server_key_version: "encryptionkey@0"
I20260925 04:43:27.600648 23116 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 04:43:27.603382 23131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:43:27.604853 23116 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 04:43:27.604979 23116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/minicluster-data/master-1/wal
uuid: "71d57ede7cf44d55bcf81e6677871726"
format_stamp: "Formatted at 2026-09-25 04:43:27 on dist-test-slave-dc95"
server_key: "bd09bd2b780e20f4af1d24f612216d5f"
server_key_iv: "8c724dba84b5223af88d74fe4e6d8df4"
server_key_version: "encryptionkey@0"
I20260925 04:43:27.605114 23116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:43:27.624951 23116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:43:27.628324 23116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:43:27.628515 23116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:43:27.635504 23116 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:46375
I20260925 04:43:27.635826 23183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:46375 every 8 connection(s)
I20260925 04:43:27.636569 23116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/minicluster-data/master-1/data/info.pb
I20260925 04:43:27.637526 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 23116
I20260925 04:43:27.637655 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/minicluster-data/master-1/wal/instance
I20260925 04:43:27.637959 392 external_mini_cluster.cc:1428] Setting key 9723970152240ade85370edc380b4775
I20260925 04:43:27.640093 23184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:43:27.644183 23184 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45775 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 25 04:43:27 dist-test-slave-dc95 krb5kdc[23028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311407, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:43:27.656641 23109 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46375 } attempt: 3
I20260925 04:43:27.660504 23112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:27.646226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38742 (local address 127.0.98.62:45775)
0925 04:43:27.646383 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:27.646388 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:27.647401 (+ 1013us) server_negotiation.cc:407] Connection header received
0925 04:43:27.648454 (+ 1053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:27.648458 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:27.648521 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:27.648629 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:27.650133 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:27.650662 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:27.652469 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:27.652745 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:27.652843 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:27.655964 (+ 3121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:43:27.655967 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:43:27.655969 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:43:27.655997 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:43:27.657569 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:27.658264 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:27.658267 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:27.658269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:27.658356 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:27.658812 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:27.658816 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:27.658818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:27.659040 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:43:27.659140 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:27.660236 (+ 1096us) server_negotiation.cc:299] Negotiation successful
0925 04:43:27.660370 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 04:43:27.661199 23187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:27.646713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45775 (local address 127.0.0.1:38742)
0925 04:43:27.647235 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0925 04:43:27.647281 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 04:43:27.648214 (+ 933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:43:27.648793 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:43:27.648799 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:43:27.649223 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:43:27.649949 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:43:27.649971 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:43:27.650802 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:43:27.650806 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:43:27.652304 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:43:27.652318 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:43:27.652928 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:43:27.652934 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:43:27.653161 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:27.653836 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:43:27.653860 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:43:27.655750 (+ 1890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:43:27.657659 (+ 1909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:43:27.657665 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:43:27.657680 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:43:27.658132 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:43:27.658481 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:43:27.658485 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:43:27.658487 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:43:27.658689 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:43:27.659162 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:43:27.659168 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:43:27.659468 (+ 300us) client_negotiation.cc:769] Sending connection context
0925 04:43:27.659690 (+ 222us) client_negotiation.cc:240] Negotiation successful
0925 04:43:27.660058 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":129,"threads_started":1}
I20260925 04:43:27.662657 23184 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46375 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 04:43:27 dist-test-slave-dc95 krb5kdc[23028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311407, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260925 04:43:27.667557 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskilA2AO/build/debug/bin/kudu
/tmp/dist-test-taskilA2AO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:42351
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45775,127.0.98.61:46375,127.0.98.60:42351
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:43:27.667951 23111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:27.657142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46375 (local address 127.0.0.1:40210)
0925 04:43:27.657364 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0925 04:43:27.657379 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:43:27.657493 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:43:27.658058 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:43:27.658061 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:43:27.658188 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:43:27.658350 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:43:27.658355 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:43:27.659702 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:43:27.659706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:43:27.660488 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:43:27.660501 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:43:27.661305 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:43:27.661310 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:43:27.661429 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:27.661973 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:43:27.661988 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:43:27.663781 (+ 1793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:43:27.665872 (+ 2091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:43:27.665876 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:43:27.665878 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:43:27.666155 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:43:27.666505 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:43:27.666508 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:43:27.666510 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:43:27.666906 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:43:27.667386 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:43:27.667390 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:43:27.667497 (+ 107us) client_negotiation.cc:769] Sending connection context
0925 04:43:27.667598 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 04:43:27.667775 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128}
Sep 25 04:43:27 dist-test-slave-dc95 krb5kdc[23028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311407, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 04:43:27.668303 23189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:27.657244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40210 (local address 127.0.98.61:46375)
0925 04:43:27.657556 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:27.657567 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:27.657591 (+ 24us) server_negotiation.cc:407] Connection header received
0925 04:43:27.657638 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:27.657644 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:27.657889 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:27.658137 (+ 248us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:27.658505 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:27.659499 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:27.660637 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:27.661166 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:27.661281 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:27.663909 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:43:27.663918 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:43:27.663929 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:43:27.663951 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:43:27.665772 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:27.666282 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:27.666288 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:27.666290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:27.666398 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:27.667004 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:27.667008 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:27.667010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:27.667280 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:43:27.667406 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:27.667738 (+ 332us) server_negotiation.cc:299] Negotiation successful
0925 04:43:27.667891 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":98,"threads_started":1}
I20260925 04:43:27.670322 23109 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42351 } has no permanent_uuid. Determining permanent_uuid...
I20260925 04:43:27.671096 23111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:27.670830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42351 (local address 127.0.0.1:52850)
0925 04:43:27.670956 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 04:43:27.671014 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260925 04:43:27.671777 23111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:27.671516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42351 (local address 127.0.0.1:52852)
0925 04:43:27.671671 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 04:43:27.671717 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260925 04:43:27.671883 23187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:27.663163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46375 (local address 127.0.0.1:40214)
0925 04:43:27.663264 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0925 04:43:27.663277 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:43:27.663354 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:43:27.663815 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:43:27.663818 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:43:27.663996 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:43:27.664079 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:43:27.664085 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:43:27.664862 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:43:27.664865 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:43:27.665637 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:43:27.665649 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:43:27.666287 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:43:27.666291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:43:27.666395 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:27.666866 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:43:27.666880 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:43:27.668321 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:43:27.670019 (+ 1698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:43:27.670022 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:43:27.670024 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:43:27.670197 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:43:27.670824 (+ 627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:43:27.670827 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:43:27.670829 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:43:27.670909 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:43:27.671348 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:43:27.671350 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:43:27.671421 (+ 71us) client_negotiation.cc:769] Sending connection context
0925 04:43:27.671501 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 04:43:27.671716 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
W20260925 04:43:27.671895 23109 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42351: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42351: connect: Connection refused (error 111)
I20260925 04:43:27.672397 23190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:27.663266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40214 (local address 127.0.98.61:46375)
0925 04:43:27.663587 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:27.663596 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:27.663633 (+ 37us) server_negotiation.cc:407] Connection header received
0925 04:43:27.663667 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:27.663670 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:27.663714 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:27.663847 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:27.664174 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:27.664767 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:27.665758 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:27.666160 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:27.666259 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:27.668483 (+ 2224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:43:27.668487 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:43:27.668489 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:43:27.668509 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:43:27.669919 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:27.670660 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:27.670663 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:27.670664 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:27.670737 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:27.670999 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:27.671003 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:27.671004 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:27.671230 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:43:27.671317 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:27.672123 (+ 806us) server_negotiation.cc:299] Negotiation successful
0925 04:43:27.672236 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":129,"threads_started":1}
I20260925 04:43:27.673046 23184 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42351 } has no permanent_uuid. Determining permanent_uuid...
I20260925 04:43:27.673767 23187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:27.673490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42351 (local address 127.0.0.1:52858)
0925 04:43:27.673621 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 04:43:27.673709 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 04:43:27.674014 23134 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42351: connect: Connection refused (error 111)
I20260925 04:43:27.674916 23187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:27.674719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42351 (local address 127.0.0.1:52862)
0925 04:43:27.674822 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0925 04:43:27.674865 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 04:43:27.675055 23184 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42351: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42351: connect: Connection refused (error 111)
I20260925 04:43:27.703089 23109 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42351 } attempt: 1
I20260925 04:43:27.704085 23111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:27.703748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42351 (local address 127.0.0.1:52868)
0925 04:43:27.703931 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 04:43:27.703999 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260925 04:43:27.704815 23111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:27.704603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42351 (local address 127.0.0.1:52874)
0925 04:43:27.704706 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0925 04:43:27.704754 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 04:43:27.704929 23109 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42351: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42351: connect: Connection refused (error 111)
I20260925 04:43:27.724308 23184 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42351 } attempt: 1
I20260925 04:43:27.725171 23187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:27.724861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42351 (local address 127.0.0.1:52890)
0925 04:43:27.725034 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 04:43:27.725093 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260925 04:43:27.725993 23187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:27.725783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42351 (local address 127.0.0.1:52906)
0925 04:43:27.725884 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0925 04:43:27.725936 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 04:43:27.726110 23184 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42351: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42351: connect: Connection refused (error 111)
I20260925 04:43:27.780107 23109 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42351 } attempt: 2
I20260925 04:43:27.781049 23111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:27.780743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42351 (local address 127.0.0.1:52918)
0925 04:43:27.780898 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 04:43:27.780962 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 04:43:27.781872 23111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:27.781608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42351 (local address 127.0.0.1:52926)
0925 04:43:27.781726 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 04:43:27.781790 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 04:43:27.782112 23109 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42351: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42351: connect: Connection refused (error 111)
W20260925 04:43:27.786962 23191 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 04:43:27.787292 23191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:43:27.787336 23191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:43:27.787402 23191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:43:27.791795 23191 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:43:27.791882 23191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:43:27.791908 23191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:43:27.791940 23191 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:43:27.791970 23191 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:43:27.794265 23184 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42351 } attempt: 2
I20260925 04:43:27.795166 23187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:27.794837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42351 (local address 127.0.0.1:52934)
0925 04:43:27.795015 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 04:43:27.795083 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 04:43:27.795984 23187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:27.795742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42351 (local address 127.0.0.1:52938)
0925 04:43:27.795873 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 04:43:27.795921 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 04:43:27.796108 23184 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42351: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42351: connect: Connection refused (error 111)
I20260925 04:43:27.798228 23191 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45775,127.0.98.61:46375,127.0.98.60:42351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:42351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:43:27.799389 23191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:43:27.800519 23191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:43:27.809231 23198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:43:27.809255 23197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:43:27.809667 23200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:43:27.809782 23191 server_base.cc:1034] running on GCE node
I20260925 04:43:27.810309 23191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:43:27.811488 23191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:43:27.812669 23191 hybrid_clock.cc:648] HybridClock initialized: now 1790311407812639 us; error 45 us; skew 500 ppm
Sep 25 04:43:27 dist-test-slave-dc95 krb5kdc[23028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311407, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:43:27.815575 23191 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260925 04:43:27.816636 23191 webserver.cc:460] Webserver started at http://127.0.98.60:35409/ using document root <none> and password file <none>
I20260925 04:43:27.817186 23191 fs_manager.cc:359] Metadata directory not provided
I20260925 04:43:27.817247 23191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:43:27.817420 23191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:43:27.819577 23191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/minicluster-data/master-2/data/instance:
uuid: "52475087907040b6a8c586c99b26d844"
format_stamp: "Formatted at 2026-09-25 04:43:27 on dist-test-slave-dc95"
server_key: "6b0aedcdd3833bc53e35c8b423393e4b"
server_key_iv: "b9f40052b78376a2681199f1e787fb83"
server_key_version: "encryptionkey@0"
I20260925 04:43:27.820176 23191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/minicluster-data/master-2/wal/instance:
uuid: "52475087907040b6a8c586c99b26d844"
format_stamp: "Formatted at 2026-09-25 04:43:27 on dist-test-slave-dc95"
server_key: "6b0aedcdd3833bc53e35c8b423393e4b"
server_key_iv: "b9f40052b78376a2681199f1e787fb83"
server_key_version: "encryptionkey@0"
I20260925 04:43:27.824448 23191 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 04:43:27.827095 23207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:43:27.828454 23191 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:43:27.828573 23191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/minicluster-data/master-2/wal
uuid: "52475087907040b6a8c586c99b26d844"
format_stamp: "Formatted at 2026-09-25 04:43:27 on dist-test-slave-dc95"
server_key: "6b0aedcdd3833bc53e35c8b423393e4b"
server_key_iv: "b9f40052b78376a2681199f1e787fb83"
server_key_version: "encryptionkey@0"
I20260925 04:43:27.828672 23191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:43:27.876426 23191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:43:27.879853 23191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:43:27.880054 23191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:43:27.881304 23109 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42351 } attempt: 3
I20260925 04:43:27.882196 23111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:27.881831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42351 (local address 127.0.0.1:52952)
0925 04:43:27.882002 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 04:43:27.882079 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260925 04:43:27.883040 23111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:27.882763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42351 (local address 127.0.0.1:52958)
0925 04:43:27.882892 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 04:43:27.882954 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 04:43:27.883239 23109 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42351: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42351: connect: Connection refused (error 111)
I20260925 04:43:27.887267 23184 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42351 } attempt: 3
I20260925 04:43:27.887672 23191 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:42351
I20260925 04:43:27.888007 23259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:42351 every 8 connection(s)
I20260925 04:43:27.888722 23191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/minicluster-data/master-2/data/info.pb
I20260925 04:43:27.892149 23260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 04:43:27 dist-test-slave-dc95 krb5kdc[23028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311407, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 04:43:27.896764 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskilA2AO/build/debug/bin/kudu as pid 23191
I20260925 04:43:27.896911 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskilA2AO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311123478858-392-0/minicluster-data/master-2/wal/instance
I20260925 04:43:27.897167 392 external_mini_cluster.cc:1428] Setting key 4120c7e7f9a911ef141fe29e09131461
I20260925 04:43:27.896947 23260 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45775 } has no permanent_uuid. Determining permanent_uuid...
I20260925 04:43:27.901283 23187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:27.887867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42351 (local address 127.0.0.1:52966)
0925 04:43:27.888017 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 04:43:27.888035 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 04:43:27.888141 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:43:27.891174 (+ 3033us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:43:27.891178 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:43:27.891364 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:43:27.891483 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:43:27.891491 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:43:27.893338 (+ 1847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:43:27.893342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:43:27.893931 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:43:27.893944 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:43:27.894923 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:43:27.894926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:43:27.894998 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:27.895504 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:43:27.895515 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:43:27.896854 (+ 1339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:43:27.899268 (+ 2414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:43:27.899271 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:43:27.899274 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:43:27.899543 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:43:27.899967 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:43:27.899971 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:43:27.899973 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:43:27.900082 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:43:27.900753 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:43:27.900756 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:43:27.900865 (+ 109us) client_negotiation.cc:769] Sending connection context
0925 04:43:27.900957 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 04:43:27.901107 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 04:43:27.902454 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:27.888866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52966 (local address 127.0.98.60:42351)
0925 04:43:27.889458 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:27.889477 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:27.889521 (+ 44us) server_negotiation.cc:407] Connection header received
0925 04:43:27.890372 (+ 851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:27.890396 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:27.890846 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:27.891219 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:27.891665 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:27.893149 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:27.894083 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:27.894793 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:27.894966 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:27.897127 (+ 2161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:43:27.897133 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:43:27.897151 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:43:27.897186 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:43:27.899130 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:27.899705 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:27.899711 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:27.899717 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:27.899874 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:27.900182 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:27.900185 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:27.900187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:27.900646 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:43:27.900761 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:27.901131 (+ 370us) server_negotiation.cc:299] Negotiation successful
0925 04:43:27.901437 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":136,"threads_started":1}
Sep 25 04:43:27 dist-test-slave-dc95 krb5kdc[23028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311407, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 25 04:43:27 dist-test-slave-dc95 krb5kdc[23028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 04:43:27.907018 23184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71d57ede7cf44d55bcf81e6677871726: Bootstrap starting.
I20260925 04:43:27.909756 23184 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71d57ede7cf44d55bcf81e6677871726: Neither blocks nor log segments found. Creating new log.
I20260925 04:43:27.910620 23112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:27.899186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38750 (local address 127.0.98.62:45775)
0925 04:43:27.899319 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:27.899324 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:27.900030 (+ 706us) server_negotiation.cc:407] Connection header received
0925 04:43:27.900178 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:27.900181 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:27.900231 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:27.900311 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:27.901108 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:27.901802 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:27.902894 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:27.903202 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:27.903279 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:27.906197 (+ 2918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:43:27.906201 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:43:27.906204 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:43:27.906238 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:43:27.908078 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:27.909126 (+ 1048us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:27.909130 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:27.909132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:27.909219 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:27.909536 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:27.909540 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:27.909542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:27.909789 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:43:27.909880 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:27.910317 (+ 437us) server_negotiation.cc:299] Negotiation successful
0925 04:43:27.910449 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 04:43:27.910636 23184 log.cc:824] T 00000000000000000000000000000000 P 71d57ede7cf44d55bcf81e6677871726: Log is configured to *not* fsync() on all Append() calls
I20260925 04:43:27.910801 23265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:27.899120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45775 (local address 127.0.0.1:38750)
0925 04:43:27.899921 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0925 04:43:27.899972 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 04:43:27.900079 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:43:27.900359 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:43:27.900365 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:43:27.900798 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:43:27.900959 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:43:27.900967 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:43:27.901927 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:43:27.901929 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:43:27.902743 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:43:27.902754 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:43:27.903434 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:43:27.903437 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:43:27.903549 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:27.904096 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:43:27.904113 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:43:27.906047 (+ 1934us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:43:27.908743 (+ 2696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:43:27.908754 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:43:27.908759 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:43:27.909026 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:43:27.909309 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:43:27.909312 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:43:27.909315 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:43:27.909417 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:43:27.909959 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:43:27.909965 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:43:27.910078 (+ 113us) client_negotiation.cc:769] Sending connection context
0925 04:43:27.910248 (+ 170us) client_negotiation.cc:240] Negotiation successful
0925 04:43:27.910450 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":147,"threads_started":1}
I20260925 04:43:27.911334 23264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:27.897796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52976 (local address 127.0.98.60:42351)
0925 04:43:27.898126 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:27.898135 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:27.898398 (+ 263us) server_negotiation.cc:407] Connection header received
0925 04:43:27.898528 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:27.898531 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:27.898579 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:27.898649 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:27.899534 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:27.900281 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:27.901691 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:27.902057 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:27.902159 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:27.906276 (+ 4117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:43:27.906280 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:43:27.906282 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:43:27.906306 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:43:27.908626 (+ 2320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:27.909316 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:27.909319 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:27.909321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:27.909466 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:27.909817 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:27.909820 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:27.909822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:27.910037 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:43:27.910209 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:27.911093 (+ 884us) server_negotiation.cc:299] Negotiation successful
0925 04:43:27.911195 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"spinlock_wait_cycles":896,"thread_start_us":83,"threads_started":1}
I20260925 04:43:27.912159 23260 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46375 } has no permanent_uuid. Determining permanent_uuid...
I20260925 04:43:27.913246 23184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71d57ede7cf44d55bcf81e6677871726: No bootstrap required, opened a new log
Sep 25 04:43:27 dist-test-slave-dc95 krb5kdc[23028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311407, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 04:43:27.917649 23184 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71d57ede7cf44d55bcf81e6677871726 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ae38ec601d45f690d0d89d0bb754c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45775 } } peers { permanent_uuid: "71d57ede7cf44d55bcf81e6677871726" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46375 } } peers { permanent_uuid: "52475087907040b6a8c586c99b26d844" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42351 } }
I20260925 04:43:27.917912 23184 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71d57ede7cf44d55bcf81e6677871726 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71d57ede7cf44d55bcf81e6677871726, State: Initialized, Role: FOLLOWER
Sep 25 04:43:27 dist-test-slave-dc95 krb5kdc[23028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 04:43:27.918290 23184 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71d57ede7cf44d55bcf81e6677871726 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ae38ec601d45f690d0d89d0bb754c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45775 } } peers { permanent_uuid: "71d57ede7cf44d55bcf81e6677871726" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46375 } } peers { permanent_uuid: "52475087907040b6a8c586c99b26d844" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42351 } }
I20260925 04:43:27.919113 23267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71d57ede7cf44d55bcf81e6677871726 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ae38ec601d45f690d0d89d0bb754c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45775 } } peers { permanent_uuid: "71d57ede7cf44d55bcf81e6677871726" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46375 } } peers { permanent_uuid: "52475087907040b6a8c586c99b26d844" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42351 } } }
I20260925 04:43:27.919270 23267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71d57ede7cf44d55bcf81e6677871726 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:43:27.920516 23184 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71d57ede7cf44d55bcf81e6677871726 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:43:27.921787 23265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:27.912625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46375 (local address 127.0.0.1:40218)
0925 04:43:27.912871 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0925 04:43:27.912885 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:43:27.912959 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:43:27.913181 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:43:27.913183 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:43:27.913410 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:43:27.913533 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:43:27.913539 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:43:27.914632 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:43:27.914636 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:43:27.915447 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:43:27.915458 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:43:27.916065 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:43:27.916069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:43:27.916165 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:27.916587 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:43:27.916601 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:43:27.918031 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:43:27.920156 (+ 2125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:43:27.920160 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:43:27.920162 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:43:27.920409 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:43:27.920693 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:43:27.920696 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:43:27.920698 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:43:27.920781 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:43:27.921254 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:43:27.921256 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:43:27.921352 (+ 96us) client_negotiation.cc:769] Sending connection context
0925 04:43:27.921461 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 04:43:27.921607 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171}
I20260925 04:43:27.922382 23190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:27.912866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40218 (local address 127.0.98.61:46375)
0925 04:43:27.912979 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:27.912983 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:27.912996 (+ 13us) server_negotiation.cc:407] Connection header received
0925 04:43:27.913034 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:27.913037 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:27.913077 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:27.913155 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:27.913666 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:27.914506 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:27.915580 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:27.915943 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:27.916050 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:27.918212 (+ 2162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:43:27.918217 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:43:27.918220 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:43:27.918253 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:43:27.920025 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:27.920499 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:27.920503 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:27.920505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:27.920614 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:27.920863 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:27.920866 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:27.920868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:27.921160 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:43:27.921246 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:27.922168 (+ 922us) server_negotiation.cc:299] Negotiation successful
0925 04:43:27.922272 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":48}
I20260925 04:43:27.922524 23112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:27.912654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38752 (local address 127.0.98.62:45775)
0925 04:43:27.912760 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:27.912764 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:27.912775 (+ 11us) server_negotiation.cc:407] Connection header received
0925 04:43:27.912809 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:27.912812 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:27.912854 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:27.912924 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:27.913694 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:27.914507 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:27.915309 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:27.915534 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:27.915672 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:27.918763 (+ 3091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:43:27.918767 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:43:27.918770 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:43:27.918798 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:43:27.920219 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:27.920723 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:27.920727 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:27.920729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:27.920817 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:27.921092 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:27.921095 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:27.921097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:27.921335 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:43:27.922158 (+ 823us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:27.922235 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 04:43:27.922370 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 04:43:27.923070 23260 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42351 } has no permanent_uuid. Determining permanent_uuid...
W20260925 04:43:27.926390 23278 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 71d57ede7cf44d55bcf81e6677871726: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 04:43:27.926544 23278 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 04:43:27.924553 23116 master_runner.cc:186] Error getting master registration for 127.0.98.62:45775: OK, instance_id { permanent_uuid: "05ae38ec601d45f690d0d89d0bb754c9" instance_seqno: 1790311407409212 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 04:43:27.927857 23116 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 25 04:43:27 dist-test-slave-dc95 krb5kdc[23028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311407, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 25 04:43:27 dist-test-slave-dc95 krb5kdc[23028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 04:43:27.933243 23264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:27.923915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52988 (local address 127.0.98.60:42351)
0925 04:43:27.924026 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:27.924030 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:27.924140 (+ 110us) server_negotiation.cc:407] Connection header received
0925 04:43:27.924266 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:27.924268 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:27.924302 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:27.924341 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:27.924801 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:27.925541 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:27.926499 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:27.926820 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:27.926893 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:27.929377 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:43:27.929381 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:43:27.929383 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:43:27.929404 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:43:27.931006 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:27.931465 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:27.931469 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:27.931471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:27.931556 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:27.932329 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:27.932333 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:27.932335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:27.932603 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:43:27.932688 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:27.932947 (+ 259us) server_negotiation.cc:299] Negotiation successful
0925 04:43:27.933082 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 04:43:27.933665 23265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:27.923656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42351 (local address 127.0.0.1:52988)
0925 04:43:27.924085 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0925 04:43:27.924097 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:43:27.924189 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:43:27.924449 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:43:27.924451 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:43:27.924607 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:43:27.924698 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:43:27.924704 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:43:27.925654 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:43:27.925656 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:43:27.926380 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:43:27.926391 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:43:27.927029 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:43:27.927032 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:43:27.927101 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:27.927794 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:43:27.927808 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:43:27.929249 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:43:27.931103 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:43:27.931107 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:43:27.931110 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:43:27.931363 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:43:27.932129 (+ 766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:43:27.932132 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:43:27.932135 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:43:27.932233 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:43:27.932688 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:43:27.932691 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:43:27.932778 (+ 87us) client_negotiation.cc:769] Sending connection context
0925 04:43:27.933408 (+ 630us) client_negotiation.cc:240] Negotiation successful
0925 04:43:27.933535 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384}
I20260925 04:43:27.934046 23190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:27.923821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40228 (local address 127.0.98.61:46375)
0925 04:43:27.924134 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:27.924138 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:27.924149 (+ 11us) server_negotiation.cc:407] Connection header received
0925 04:43:27.924180 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:27.924183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:27.924223 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:27.925017 (+ 794us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:27.925101 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:27.925797 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:27.926878 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:27.927191 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:27.927577 (+ 386us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:27.930052 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:43:27.930056 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:43:27.930059 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:43:27.930087 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:43:27.931548 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:27.932050 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:27.932053 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:27.932054 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:27.932127 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:27.932473 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:27.932476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:27.932478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:27.932708 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:43:27.932942 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:27.933316 (+ 374us) server_negotiation.cc:299] Negotiation successful
0925 04:43:27.933513 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246}
I20260925 04:43:27.936954 23260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52475087907040b6a8c586c99b26d844: Bootstrap starting.
I20260925 04:43:27.937789 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:43:27.939389 23260 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52475087907040b6a8c586c99b26d844: Neither blocks nor log segments found. Creating new log.
I20260925 04:43:27.940176 23260 log.cc:824] T 00000000000000000000000000000000 P 52475087907040b6a8c586c99b26d844: Log is configured to *not* fsync() on all Append() calls
I20260925 04:43:27.939272 23083 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "05ae38ec601d45f690d0d89d0bb754c9"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38752
I20260925 04:43:27.942227 23260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52475087907040b6a8c586c99b26d844: No bootstrap required, opened a new log
I20260925 04:43:27.945353 23260 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52475087907040b6a8c586c99b26d844 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ae38ec601d45f690d0d89d0bb754c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45775 } } peers { permanent_uuid: "71d57ede7cf44d55bcf81e6677871726" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46375 } } peers { permanent_uuid: "52475087907040b6a8c586c99b26d844" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42351 } }
I20260925 04:43:27.945577 23260 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52475087907040b6a8c586c99b26d844 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52475087907040b6a8c586c99b26d844, State: Initialized, Role: FOLLOWER
I20260925 04:43:27.945887 23260 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52475087907040b6a8c586c99b26d844 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ae38ec601d45f690d0d89d0bb754c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45775 } } peers { permanent_uuid: "71d57ede7cf44d55bcf81e6677871726" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46375 } } peers { permanent_uuid: "52475087907040b6a8c586c99b26d844" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42351 } }
I20260925 04:43:27.946483 23281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52475087907040b6a8c586c99b26d844 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ae38ec601d45f690d0d89d0bb754c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45775 } } peers { permanent_uuid: "71d57ede7cf44d55bcf81e6677871726" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46375 } } peers { permanent_uuid: "52475087907040b6a8c586c99b26d844" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42351 } } }
I20260925 04:43:27.946729 23281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52475087907040b6a8c586c99b26d844 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:43:27.947081 23260 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52475087907040b6a8c586c99b26d844 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 04:43:27.952433 23292 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 52475087907040b6a8c586c99b26d844: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 04:43:27.952592 23292 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 04:43:27.950668 23191 master_runner.cc:186] Error getting master registration for 127.0.98.62:45775: OK, instance_id { permanent_uuid: "05ae38ec601d45f690d0d89d0bb754c9" instance_seqno: 1790311407409212 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 04:43:27.954030 23191 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 04:43:28.047395 23109 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42351 } attempt: 4
Sep 25 04:43:28 dist-test-slave-dc95 krb5kdc[23028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311407, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 04:43:28.058410 23111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:28.048131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42351 (local address 127.0.0.1:52994)
0925 04:43:28.048314 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 04:43:28.048332 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 04:43:28.048443 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:43:28.049158 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:43:28.049162 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:43:28.049433 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:43:28.049612 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:43:28.049621 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:43:28.050642 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:43:28.050646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:43:28.051505 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:43:28.051520 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:43:28.052130 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:43:28.052134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:43:28.052224 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:28.052854 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:43:28.052872 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:43:28.054537 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:43:28.056682 (+ 2145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:43:28.056686 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:43:28.056689 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:43:28.056972 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:43:28.057287 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:43:28.057290 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:43:28.057292 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:43:28.057397 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:43:28.057838 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:43:28.057841 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:43:28.057962 (+ 121us) client_negotiation.cc:769] Sending connection context
0925 04:43:28.058071 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 04:43:28.058235 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260925 04:43:28.058435 23264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:28.048731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52994 (local address 127.0.98.60:42351)
0925 04:43:28.048904 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:28.048909 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:28.048924 (+ 15us) server_negotiation.cc:407] Connection header received
0925 04:43:28.048968 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:28.048971 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:28.049019 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:28.049102 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:28.049777 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:28.050532 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:28.051669 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:28.052002 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:28.052110 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:28.054711 (+ 2601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:43:28.054716 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:43:28.054719 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:43:28.054750 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:43:28.056529 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:28.057076 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:28.057079 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:28.057081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:28.057178 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:28.057492 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:28.057496 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:28.057497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:28.057733 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:43:28.057832 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:28.058124 (+ 292us) server_negotiation.cc:299] Negotiation successful
0925 04:43:28.058263 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 04:43:28.063189 23109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05ae38ec601d45f690d0d89d0bb754c9: Bootstrap starting.
I20260925 04:43:28.066488 23109 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05ae38ec601d45f690d0d89d0bb754c9: Neither blocks nor log segments found. Creating new log.
I20260925 04:43:28.067534 23109 log.cc:824] T 00000000000000000000000000000000 P 05ae38ec601d45f690d0d89d0bb754c9: Log is configured to *not* fsync() on all Append() calls
I20260925 04:43:28.070060 23109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05ae38ec601d45f690d0d89d0bb754c9: No bootstrap required, opened a new log
I20260925 04:43:28.073395 23109 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05ae38ec601d45f690d0d89d0bb754c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ae38ec601d45f690d0d89d0bb754c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45775 } } peers { permanent_uuid: "71d57ede7cf44d55bcf81e6677871726" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46375 } } peers { permanent_uuid: "52475087907040b6a8c586c99b26d844" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42351 } }
I20260925 04:43:28.073685 23109 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05ae38ec601d45f690d0d89d0bb754c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05ae38ec601d45f690d0d89d0bb754c9, State: Initialized, Role: FOLLOWER
I20260925 04:43:28.074045 23109 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05ae38ec601d45f690d0d89d0bb754c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ae38ec601d45f690d0d89d0bb754c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45775 } } peers { permanent_uuid: "71d57ede7cf44d55bcf81e6677871726" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46375 } } peers { permanent_uuid: "52475087907040b6a8c586c99b26d844" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42351 } }
I20260925 04:43:28.074745 23295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05ae38ec601d45f690d0d89d0bb754c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ae38ec601d45f690d0d89d0bb754c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45775 } } peers { permanent_uuid: "71d57ede7cf44d55bcf81e6677871726" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46375 } } peers { permanent_uuid: "52475087907040b6a8c586c99b26d844" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42351 } } }
I20260925 04:43:28.075083 23295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05ae38ec601d45f690d0d89d0bb754c9 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:43:28.075686 23109 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05ae38ec601d45f690d0d89d0bb754c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:43:28.075800 23083 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05ae38ec601d45f690d0d89d0bb754c9 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 04:43:28.076021 23083 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05ae38ec601d45f690d0d89d0bb754c9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:43:28.077533 23083 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05ae38ec601d45f690d0d89d0bb754c9 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ae38ec601d45f690d0d89d0bb754c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45775 } } peers { permanent_uuid: "71d57ede7cf44d55bcf81e6677871726" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46375 } } peers { permanent_uuid: "52475087907040b6a8c586c99b26d844" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42351 } }
I20260925 04:43:28.079957 23158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "05ae38ec601d45f690d0d89d0bb754c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "71d57ede7cf44d55bcf81e6677871726"
I20260925 04:43:28.080272 23158 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71d57ede7cf44d55bcf81e6677871726 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:43:28.081792 23158 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 71d57ede7cf44d55bcf81e6677871726 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05ae38ec601d45f690d0d89d0bb754c9 in term 1.
I20260925 04:43:28.082340 23234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "05ae38ec601d45f690d0d89d0bb754c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "52475087907040b6a8c586c99b26d844"
I20260925 04:43:28.082633 23234 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52475087907040b6a8c586c99b26d844 [term 0 FOLLOWER]: Advancing to term 1
W20260925 04:43:28.082783 23306 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 05ae38ec601d45f690d0d89d0bb754c9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 04:43:28.082877 23306 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 04:43:28.084352 23234 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 52475087907040b6a8c586c99b26d844 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05ae38ec601d45f690d0d89d0bb754c9 in term 1.
I20260925 04:43:28.085371 23083 leader_election.cc:288] T 00000000000000000000000000000000 P 05ae38ec601d45f690d0d89d0bb754c9 [CANDIDATE]: Term 1 election: Requested vote from peers 71d57ede7cf44d55bcf81e6677871726 (127.0.98.61:46375), 52475087907040b6a8c586c99b26d844 (127.0.98.60:42351)
I20260925 04:43:28.085506 23060 leader_election.cc:302] T 00000000000000000000000000000000 P 05ae38ec601d45f690d0d89d0bb754c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05ae38ec601d45f690d0d89d0bb754c9, 71d57ede7cf44d55bcf81e6677871726; no voters:
I20260925 04:43:28.091913 23041 master_runner.cc:263] No leader master found from master 05ae38ec601d45f690d0d89d0bb754c9
I20260925 04:43:28.092015 23041 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 04:43:28.092445 23295 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05ae38ec601d45f690d0d89d0bb754c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:43:28.092767 23295 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05ae38ec601d45f690d0d89d0bb754c9 [term 1 LEADER]: Becoming Leader. State: Replica: 05ae38ec601d45f690d0d89d0bb754c9, State: Running, Role: LEADER
I20260925 04:43:28.093118 23295 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05ae38ec601d45f690d0d89d0bb754c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ae38ec601d45f690d0d89d0bb754c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45775 } } peers { permanent_uuid: "71d57ede7cf44d55bcf81e6677871726" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46375 } } peers { permanent_uuid: "52475087907040b6a8c586c99b26d844" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42351 } }
I20260925 04:43:28.095069 23295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05ae38ec601d45f690d0d89d0bb754c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05ae38ec601d45f690d0d89d0bb754c9. Latest consensus state: current_term: 1 leader_uuid: "05ae38ec601d45f690d0d89d0bb754c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ae38ec601d45f690d0d89d0bb754c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45775 } } peers { permanent_uuid: "71d57ede7cf44d55bcf81e6677871726" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46375 } } peers { permanent_uuid: "52475087907040b6a8c586c99b26d844" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42351 } } }
I20260925 04:43:28.095192 23295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05ae38ec601d45f690d0d89d0bb754c9 [sys.catalog]: This master's current role is: LEADER
I20260925 04:43:28.097718 23314 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:43:28.098722 23314 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:43:28.115322 23264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:28.095775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53008 (local address 127.0.98.60:42351)
0925 04:43:28.095907 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:28.095913 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:28.104171 (+ 8258us) server_negotiation.cc:407] Connection header received
0925 04:43:28.104229 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:28.104232 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:28.104284 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:28.104357 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:28.105527 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:28.106334 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:28.107753 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:28.108136 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:28.108218 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:28.110767 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:43:28.110771 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:43:28.110773 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:43:28.110813 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:43:28.112813 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:28.113333 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:28.113336 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:28.113338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:28.113433 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:28.113906 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:28.113911 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:28.113913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:28.114182 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:43:28.114288 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:28.114983 (+ 695us) server_negotiation.cc:299] Negotiation successful
0925 04:43:28.115145 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 04:43:28.115923 23190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:28.096773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40236 (local address 127.0.98.61:46375)
0925 04:43:28.096884 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:28.096888 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:28.104171 (+ 7283us) server_negotiation.cc:407] Connection header received
0925 04:43:28.104229 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:28.104232 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:28.104284 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:28.104363 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:28.105526 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:28.106338 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:28.107753 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:28.108136 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:28.108229 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:28.110727 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:43:28.110731 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:43:28.110734 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:43:28.110766 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:43:28.113329 (+ 2563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:28.113957 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:28.113961 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:28.113962 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:28.114049 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:28.114391 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:28.114396 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:28.114397 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:28.114629 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:43:28.114736 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:28.115584 (+ 848us) server_negotiation.cc:299] Negotiation successful
0925 04:43:28.115767 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 04:43:28.122149 23158 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 71d57ede7cf44d55bcf81e6677871726 [term 1 FOLLOWER]: Refusing update from remote peer 05ae38ec601d45f690d0d89d0bb754c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:43:28.124848 23295 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 05ae38ec601d45f690d0d89d0bb754c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71d57ede7cf44d55bcf81e6677871726" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:43:28.126056 23112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:28.096274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38760 (local address 127.0.98.62:45775)
0925 04:43:28.096386 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:28.096391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:28.109007 (+ 12616us) server_negotiation.cc:407] Connection header received
0925 04:43:28.109234 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:28.109237 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:28.109321 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:28.109387 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:28.116804 (+ 7417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:28.117511 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:28.118576 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:28.118903 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:28.118982 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:28.122001 (+ 3019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:43:28.122006 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:43:28.122009 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:43:28.122044 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:43:28.123976 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:28.124502 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:28.124506 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:28.124508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:28.124607 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:43:28.124938 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:43:28.124942 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:43:28.124943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:43:28.125180 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:43:28.125294 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:28.125653 (+ 359us) server_negotiation.cc:299] Negotiation successful
0925 04:43:28.125797 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 04:43:28.129712 23267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71d57ede7cf44d55bcf81e6677871726 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05ae38ec601d45f690d0d89d0bb754c9. Latest consensus state: current_term: 1 leader_uuid: "05ae38ec601d45f690d0d89d0bb754c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ae38ec601d45f690d0d89d0bb754c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45775 } } peers { permanent_uuid: "71d57ede7cf44d55bcf81e6677871726" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46375 } } peers { permanent_uuid: "52475087907040b6a8c586c99b26d844" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42351 } } }
I20260925 04:43:28.129874 23267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71d57ede7cf44d55bcf81e6677871726 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 04:43:28.130338 23137 tablet.cc:2369] T 00000000000000000000000000000000 P 71d57ede7cf44d55bcf81e6677871726: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 04:43:28.131501 23213 tablet.cc:2369] T 00000000000000000000000000000000 P 52475087907040b6a8c586c99b26d844: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:43:28.132017 23234 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 52475087907040b6a8c586c99b26d844 [term 1 FOLLOWER]: Refusing update from remote peer 05ae38ec601d45f690d0d89d0bb754c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:43:28.132714 23312 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 05ae38ec601d45f690d0d89d0bb754c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52475087907040b6a8c586c99b26d844" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:43:28.136328 23312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05ae38ec601d45f690d0d89d0bb754c9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "05ae38ec601d45f690d0d89d0bb754c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ae38ec601d45f690d0d89d0bb754c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45775 } } peers { permanent_uuid: "71d57ede7cf44d55bcf81e6677871726" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46375 } } peers { permanent_uuid: "52475087907040b6a8c586c99b26d844" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42351 } } }
I20260925 04:43:28.136484 23312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05ae38ec601d45f690d0d89d0bb754c9 [sys.catalog]: This master's current role is: LEADER
I20260925 04:43:28.138405 23281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52475087907040b6a8c586c99b26d844 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05ae38ec601d45f690d0d89d0bb754c9. Latest consensus state: current_term: 1 leader_uuid: "05ae38ec601d45f690d0d89d0bb754c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ae38ec601d45f690d0d89d0bb754c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45775 } } peers { permanent_uuid: "71d57ede7cf44d55bcf81e6677871726" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46375 } } peers { permanent_uuid: "52475087907040b6a8c586c99b26d844" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42351 } } }
I20260925 04:43:28.138543 23281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52475087907040b6a8c586c99b26d844 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:43:28.138736 23267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71d57ede7cf44d55bcf81e6677871726 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "05ae38ec601d45f690d0d89d0bb754c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ae38ec601d45f690d0d89d0bb754c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45775 } } peers { permanent_uuid: "71d57ede7cf44d55bcf81e6677871726" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46375 } } peers { permanent_uuid: "52475087907040b6a8c586c99b26d844" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42351 } } }
I20260925 04:43:28.138844 23267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71d57ede7cf44d55bcf81e6677871726 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:43:28.139477 23318 mvcc.cc:205] Tried to move back new op lower bound from 7333115527656878080 to 7333115527553622016. Current Snapshot: MvccSnapshot[applied={T|T < 7333115527656878080}]
I20260925 04:43:28.139917 23319 mvcc.cc:205] Tried to move back new op lower bound from 7333115527656878080 to 7333115527553622016. Current Snapshot: MvccSnapshot[applied={T|T < 7333115527656878080}]
I20260925 04:43:28.141765 23295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05ae38ec601d45f690d0d89d0bb754c9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "05ae38ec601d45f690d0d89d0bb754c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ae38ec601d45f690d0d89d0bb754c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45775 } } peers { permanent_uuid: "71d57ede7cf44d55bcf81e6677871726" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46375 } } peers { permanent_uuid: "52475087907040b6a8c586c99b26d844" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42351 } } }
I20260925 04:43:28.141911 23295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05ae38ec601d45f690d0d89d0bb754c9 [sys.catalog]: This master's current role is: LEADER
I20260925 04:43:28.143987 23281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52475087907040b6a8c586c99b26d844 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "05ae38ec601d45f690d0d89d0bb754c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ae38ec601d45f690d0d89d0bb754c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45775 } } peers { permanent_uuid: "71d57ede7cf44d55bcf81e6677871726" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46375 } } peers { permanent_uuid: "52475087907040b6a8c586c99b26d844" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42351 } } }
I20260925 04:43:28.144109 23281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52475087907040b6a8c586c99b26d844 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:43:28.145709 23314 catalog_manager.cc:1347] Generated new cluster ID: d24ccb4d808b42a7b8059194616e15bc
I20260925 04:43:28.145784 23314 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:43:28.168185 23314 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:43:28.169539 23314 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:43:28.177316 23314 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05ae38ec601d45f690d0d89d0bb754c9: Generated new TSK 0
I20260925 04:43:28.178364 23314 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 04:43:28.207703 23112 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:43:28.205982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38766 (local address 127.0.98.62:45775)
0925 04:43:28.206122 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:28.206128 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:28.206573 (+ 445us) server_negotiation.cc:407] Connection header received
0925 04:43:28.206771 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:28.206775 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:28.206829 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:28.206880 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:28.207522 (+ 642us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38766: BlockingRecv error: recv got EOF from 127.0.0.1:38766 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260925 04:43:28.207721 23264 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:43:28.206063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53016 (local address 127.0.98.60:42351)
0925 04:43:28.206187 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:28.206191 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:28.206483 (+ 292us) server_negotiation.cc:407] Connection header received
0925 04:43:28.206693 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:28.206697 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:28.206752 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:28.206820 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:28.207551 (+ 731us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53016: BlockingRecv error: recv got EOF from 127.0.0.1:53016 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260925 04:43:28.208269 23190 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:43:28.206648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40242 (local address 127.0.98.61:46375)
0925 04:43:28.206768 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:28.206772 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:28.207210 (+ 438us) server_negotiation.cc:407] Connection header received
0925 04:43:28.207345 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:28.207348 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:28.207405 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:28.207459 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:28.208143 (+ 684us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40242: BlockingRecv error: recv got EOF from 127.0.0.1:40242 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260925 04:43:28.211799 23190 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:43:28.210673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40256 (local address 127.0.98.61:46375)
0925 04:43:28.210803 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:28.210807 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:28.210863 (+ 56us) server_negotiation.cc:407] Connection header received
0925 04:43:28.211051 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:28.211053 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:28.211122 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:28.211244 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:28.211689 (+ 445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40256: BlockingRecv error: recv got EOF from 127.0.0.1:40256 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260925 04:43:28.215900 23190 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:43:28.214991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40270 (local address 127.0.98.61:46375)
0925 04:43:28.215101 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:28.215104 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:28.215116 (+ 12us) server_negotiation.cc:407] Connection header received
0925 04:43:28.215163 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:28.215165 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:28.215207 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:28.215282 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:28.215778 (+ 496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40270: BlockingRecv error: recv got EOF from 127.0.0.1:40270 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260925 04:43:28.221514 23190 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:43:28.220227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40278 (local address 127.0.98.61:46375)
0925 04:43:28.220411 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:28.220415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:28.220635 (+ 220us) server_negotiation.cc:407] Connection header received
0925 04:43:28.220741 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:28.220744 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:28.220808 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:28.220901 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:28.221386 (+ 485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40278: BlockingRecv error: recv got EOF from 127.0.0.1:40278 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
W20260925 04:43:28.229360 23190 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:43:28.228160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40292 (local address 127.0.98.61:46375)
0925 04:43:28.228355 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:28.228360 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:28.228408 (+ 48us) server_negotiation.cc:407] Connection header received
0925 04:43:28.228551 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:28.228555 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:28.228619 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:28.228743 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:28.229232 (+ 489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40292: BlockingRecv error: recv got EOF from 127.0.0.1:40292 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
W20260925 04:43:28.240537 23190 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:43:28.239415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40300 (local address 127.0.98.61:46375)
0925 04:43:28.239556 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:28.239561 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:28.239651 (+ 90us) server_negotiation.cc:407] Connection header received
0925 04:43:28.239837 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:28.239840 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:28.239901 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:28.239999 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:28.240404 (+ 405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40300: BlockingRecv error: recv got EOF from 127.0.0.1:40300 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260925 04:43:28.260097 23190 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:43:28.258893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40304 (local address 127.0.98.61:46375)
0925 04:43:28.259096 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:28.259101 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:28.259210 (+ 109us) server_negotiation.cc:407] Connection header received
0925 04:43:28.259368 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:28.259370 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:28.259419 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:28.259492 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:28.259998 (+ 506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40304: BlockingRecv error: recv got EOF from 127.0.0.1:40304 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
W20260925 04:43:28.295477 23190 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:43:28.294320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40320 (local address 127.0.98.61:46375)
0925 04:43:28.294485 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:28.294489 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:28.294586 (+ 97us) server_negotiation.cc:407] Connection header received
0925 04:43:28.294743 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:28.294747 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:28.294796 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:28.294865 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:28.295369 (+ 504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40320: BlockingRecv error: recv got EOF from 127.0.0.1:40320 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260925 04:43:28.363013 23190 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:43:28.361812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40328 (local address 127.0.98.61:46375)
0925 04:43:28.362004 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:28.362009 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:28.362111 (+ 102us) server_negotiation.cc:407] Connection header received
0925 04:43:28.362288 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:28.362291 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:28.362342 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:28.362424 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:28.362909 (+ 485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40328: BlockingRecv error: recv got EOF from 127.0.0.1:40328 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260925 04:43:28.494494 23190 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:43:28.493295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40340 (local address 127.0.98.61:46375)
0925 04:43:28.493460 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:28.493464 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:28.493566 (+ 102us) server_negotiation.cc:407] Connection header received
0925 04:43:28.493738 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:28.493741 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:28.493789 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:28.493860 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:28.494387 (+ 527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40340: BlockingRecv error: recv got EOF from 127.0.0.1:40340 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260925 04:43:28.754045 23190 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:43:28.752850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40274 (local address 127.0.98.61:46375)
0925 04:43:28.752994 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:28.752998 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:28.753145 (+ 147us) server_negotiation.cc:407] Connection header received
0925 04:43:28.753335 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:28.753337 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:28.753388 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:28.753455 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:43:28.753918 (+ 463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40274: BlockingRecv error: recv got EOF from 127.0.0.1:40274 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 04:43:28.927963 23278 catalog_manager.cc:1259] Loaded cluster ID: d24ccb4d808b42a7b8059194616e15bc
I20260925 04:43:28.928048 23278 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 71d57ede7cf44d55bcf81e6677871726: loading cluster ID for follower catalog manager: success
I20260925 04:43:28.929780 23278 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 71d57ede7cf44d55bcf81e6677871726: acquiring CA information for follower catalog manager: success
I20260925 04:43:28.930981 23278 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 71d57ede7cf44d55bcf81e6677871726: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 04:43:28.953922 23292 catalog_manager.cc:1259] Loaded cluster ID: d24ccb4d808b42a7b8059194616e15bc
I20260925 04:43:28.954013 23292 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 52475087907040b6a8c586c99b26d844: loading cluster ID for follower catalog manager: success
I20260925 04:43:28.955749 23292 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 52475087907040b6a8c586c99b26d844: acquiring CA information for follower catalog manager: success
I20260925 04:43:28.956959 23292 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 52475087907040b6a8c586c99b26d844: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 04:43:29.272156 23392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:29.268298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40278 (local address 127.0.98.61:46375)
0925 04:43:29.268565 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:29.268570 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:29.268587 (+ 17us) server_negotiation.cc:407] Connection header received
0925 04:43:29.268709 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:29.268712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:29.268864 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:29.268964 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:43:29.269236 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:29.269767 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:29.270609 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:29.270849 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:29.270942 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:29.271252 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:43:29.271734 (+ 482us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:43:29.271845 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:29.271931 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 04:43:29.271993 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":77,"threads_started":1}
I20260925 04:43:29.279462 23398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:29.275360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44440 (local address 127.0.98.60:42351)
0925 04:43:29.275676 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:29.275680 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:29.275791 (+ 111us) server_negotiation.cc:407] Connection header received
0925 04:43:29.276010 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:29.276012 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:29.276137 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:29.276270 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:43:29.276497 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:29.277114 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:29.278068 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:29.278294 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:29.278373 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:29.278626 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:43:29.279067 (+ 441us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:43:29.279162 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:29.279243 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 04:43:29.279298 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":66,"threads_started":1}
I20260925 04:43:29.287164 23392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:29.283127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40282 (local address 127.0.98.61:46375)
0925 04:43:29.283262 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:29.283266 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:29.283457 (+ 191us) server_negotiation.cc:407] Connection header received
0925 04:43:29.283653 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:29.283656 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:29.283716 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:29.283861 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:43:29.284224 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:29.284735 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:29.286045 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:29.286303 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:29.286380 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:29.286679 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:43:29.286756 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:43:29.286819 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:29.286955 (+ 136us) server_negotiation.cc:299] Negotiation successful
0925 04:43:29.287015 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 04:43:29.287241 23405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:29.282798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38898 (local address 127.0.98.62:45775)
0925 04:43:29.283117 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:29.283122 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:29.283156 (+ 34us) server_negotiation.cc:407] Connection header received
0925 04:43:29.283248 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:29.283251 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:29.283384 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:29.283506 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:43:29.283772 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:29.284522 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:29.285493 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:29.285734 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:29.285825 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:29.286148 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:43:29.286485 (+ 337us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:43:29.286580 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:29.286942 (+ 362us) server_negotiation.cc:299] Negotiation successful
0925 04:43:29.287109 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":65,"threads_started":1}
I20260925 04:43:29.287528 23398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:43:29.282833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44444 (local address 127.0.98.60:42351)
0925 04:43:29.282966 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 04:43:29.282970 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:43:29.283403 (+ 433us) server_negotiation.cc:407] Connection header received
0925 04:43:29.283553 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:43:29.283557 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:43:29.283643 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:43:29.283688 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:43:29.284055 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:29.284806 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:29.286449 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:43:29.286726 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:43:29.286824 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:43:29.287082 (+ 258us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:43:29.287152 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:43:29.287189 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0925 04:43:29.287335 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 04:43:29.287395 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 04:43:29.290710 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 23041
I20260925 04:43:29.296469 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 23116
I20260925 04:43:29.301783 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskilA2AO/build/debug/bin/kudu with pid 23191
2026-09-25T04:43:29Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3148 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6218 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (285812 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST